elasticsearch - 如何使用grok在logstash中对日志消息进行分组？

Question

 [25-Dec-2015 08:06:45] 0:: users to chek for delete
 [25-Dec-2015 08:08:44] 0:: users to chek for delete
 [25-Dec-2015 08:10:44] 3:: users to chek for delete
 [25-Dec-2015 08:10:44] Expected response code 200, got 404

     {
         "error": {
          "errors": [
           {
            "domain": "global",
            "reason": "notFound",
            "message": "Resource Not Found: userKey"
           }
          ],
          "code": 404,
          "message": "Resource Not Found: userKey"
         }
        }

    [06-Nov-2015 19:24:19 GMT] PHP Fatal error:  Class 'Test\Test\Api\Resources\Authenticate1234' not found in /apps/test/src/Test/Test/Api/Resources/ResourceFactory.php on line 10
    [06-Nov-2015 19:24:19 GMT] PHP Stack trace:
    [06-Nov-2015 19:24:19 GMT] PHP   1. {main}() /apps/test/public/api.php:0
    [06-Nov-2015 19:24:19 GMT] PHP   2. Test\Test\Api\ApiController->handleRequest() /apps/test/public/api.php:13
    [06-Nov-2015 19:24:19 GMT] PHP   3. Test\Test\Api\Resources\ResourceFactory->create() /apps/test/src/Test/Test/Api/ApiController.php:14

上面是我的日志文件的示例。我需要过滤掉每条消息。问题在于编写过滤器。前三行是三个不同的错误。

[25-Dec-2015 08:06:45] 0::用户检查删除

[25-Dec-2015 08:06:45] 0::用户检查删除

[25-Dec-2015 08:06:45] 3::用户检查删除

第四个错误是JSON消息错误。我需要将此块与上面分开。

    [25-Dec-2015 08:10:44] Expected response code 200, got 404
     {
         "error": {
          "errors": [
           {
            "domain": "global",
            "reason": "notFound",
            "message": "Resource Not Found: userKey"
           }
          ],
          "code": 404,
          "message": "Resource Not Found: userKey"
         }
        }

第五个错误是PHP堆栈跟踪。

        [06-Nov-2015 19:24:19 GMT] PHP Fatal error:  Class 'Test\Test\Api\Resources\Authenticate1234' not found in /apps/test/src/Test/Test/Api/Resources/ResourceFactory.php on line 10
        [06-Nov-2015 19:24:19 GMT] PHP Stack trace:
        [06-Nov-2015 19:24:19 GMT] PHP   1. {main}() /apps/test/public/api.php:0
        [06-Nov-2015 19:24:19 GMT] PHP   2. Test\Test\Api\ApiController->handleRequest() /apps/test/public/api.php:13
        [06-Nov-2015 19:24:19 GMT] PHP   3. Test\Test\Api\Resources\ResourceFactory->create() /apps/test/src/Test/Test/Api/ApiController.php:14

有可能设计出符合这三个条件的过滤器吗？

Answer 1

使用 multiline 选项。例如:

filter {
    multiline {
        negate    => true
        pattern   => "^\["
        what      => "previous"
    }
}

结果应如下所示:

[06-Nov-2015 19:24:19 GMT] PHP Fatal error:  Class 'Test\Test\Api\Resources\Authenticate1234' not found in /apps/test/src/Test/Test/Api/Resources/ResourceFactory.php on line 10
PHP Stack trace:
PHP   1. {main}() /apps/test/public/api.php:0
PHP   2. Test\Test\Api\ApiController->handleRequest() /apps/test/public/api.php:13
PHP   3. Test\Test\Api\Resources\ResourceFactory->create() /apps/test/src/Test/Test/Api/ApiController.php:14