logstash - 未为多行事件添加 Filebeat 多行标记

Question

我做了什么:

我在 filebeat.yml 文件中启用了多行模式:

  multiline.pattern: '^[[:space:]]+|^Caused by:'
  multiline.negate: false
  multiline.match: after

我的示例日志文件包含多行异常:

Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

从 filebeat 日志中我可以看到这些多行异常得到了正确的解析:

2017-05-15T08:21:36-07:00 DBG  Publish: {
  "@timestamp": "2017-05-15T15:21:31.768Z",
  "beat": {
    "hostname": "WIN-UV5OA3SO3LF",
    "name": "WIN-UV5OA3SO3LF",
    "version": "5.2.0"
  },
  "input_type": "log",
  "message": "Exception in thread \"main\" java.lang.IllegalStateException: A book has a null property\n       at com.example.myproject.Author.getBookIds(Author.java:38)\n       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)\nCaused by: java.lang.NullPointerException\n       at com.example.myproject.Book.getId(Book.java:22)\n       at com.example.myproject.Author.getBookIds(Author.java:35)\n       ... 1 more",
  "offset": 409,
  "source": "C:\\Filebeat\\test\\testLog.txt",
  "type": "log"
}

2017-05-15T08:21:41-07:00 DBG  Publish: {
  "@timestamp": "2017-05-15T15:21:31.768Z",
  "beat": {
    "hostname": "WIN-UV5OA3SO3LF",
    "name": "WIN-UV5OA3SO3LF",
    "version": "5.2.0"
  },
  "input_type": "log",
  "message": "Exception in thread \"main\" java.lang.NullPointerException\n        at com.example.myproject.Book.getTitle(Book.java:16)\n        at com.example.myproject.Author.getBookTitles(Author.java:25)\n        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)",
  "offset": 669,
  "source": "C:\\Filebeat\\test\\testLog.txt",
  "type": "log"
}

问题:

• filebeat 解析的多行事件未被标记为多行。
• 因此，我在 logstash 上的 grok 过滤器无法相应地处理它

    if "multiline" in [tags] {
      grok {
        match => ["message", "%{JAVASTACKTRACEPART}"]
      }
    }

任何指点将不胜感激!

Answer 1

抱歉，我不回答问题，相反我建议从另一个角度看问题:为什么不在每一行新日志的开头引入固定模式(这是常见的做法)？

这允许使用 negate -mode for multiline-feature 这是更简单和更直接的方式。

让我们接受每一行都以日期开头，如下所示:

2019.10.23 01:01:01.384500 [ERROR] Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more
2019.10.23 01:02:01.384500 [ERROR] Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

现在，只需要定义开始每一行的模式并打开否定模式:

filebeat.inputs:
- type: log
  enabled: true
  # https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
  # This regexp describes the begin of log row (for example, '2019.10.23 01:01:01.384500 ..').
  multiline.pattern: '^\d{4}.\d{2}.\d{2} \d{2}:\d{2}:\d{2}'
  multiline.negate: true
  multiline.match: after  
  paths:
    - ..

processors:
   - ..
   - dissect:
      # tokenizer syntax: https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html.
      tokenizer: "%{timestamp} [%{level}] %{?message}"
      # https://www.elastic.co/guide/en/beats/filebeat/master/dissect.html
      field: "message"
      target_prefix: ""