json - Fluentbit嵌套json解析

Question

我有以下日志

{
  "log": {
    "header": {
      "key": "value",
      "nested": "{\"key1\":\"value\",\"key2\":\"value\"}",
      "dateTime": "2019-05-08T20:58:06+00:00"
    },
    "body": {
      "path": "/request/path/",
      "method": "POST",
      "ua": "curl/7.54.0",
      "resp": 200
    }
  }
}

我正在尝试使用 fluentbit 聚合日志，我希望整个记录都是 JSON。具体问题是“log.header.nested”字段，是一个JSON字符串。我如何解析该字符串并将其替换为它的内容？

我尝试使用来自 fluentbit 的解析器过滤器.但我对 key_name 有疑问，它不适用于嵌套的 json 值。我尝试使用非嵌套字段在本地对其进行测试，并且以下配置有效:

[INPUT]
    name             tail
    path             nst.log
    read_from_head   true
    Parser           json
[FILTER]
    name          parser
    Match         *
    Parser        json
    key_name      log
    Reserve_Data  On

[FILTER]
    name          parser
    Match         *
    Parser        json
    key_name      nested
    Reserve_Data  On

[OUTPUT]
    name             stdout
    match            *

但是当我为嵌套值尝试这个过滤器时:

[FILTER]
    name          parser
    Match         *
    Parser        json
    key_name      log.header.nested
    Reserve_Data  On

它不起作用，fluentbit 文档中没有关于如何在 key_name 文件中使用嵌套键的内容。所以我尝试了:

• log.header.nested
• log_header_nest
• log['header']['nest']
• 日志[header][nest]

为了清楚起见，我希望 fluentbit 输出的日志看起来像这样:

{
  "log": {
    "header": {
      "key": "value",
      "nested": {
          "key1": "value",
          "key2": "value"
      },
      "dateTime": "2019-05-08T20:58:06+00:00"
    },
    "body": {
      "path": "/request/path/",
      "method": "POST",
      "ua": "curl/7.54.0",
      "resp": 200
    }
  }
}

Answer 1

您可以尝试将 Nest 过滤器插件与 Parser 过滤器插件结合使用。例如，我设法使用以下配置在第一级解析嵌套的 json:

    [FILTER]
        Name               nest
        Match              application.*
        Operation          lift
        Nested_under       log_processed
        Add_prefix         log_
        Wildcard           message
   [FILTER]
       Name                parser
       Match               application.*
       Key_Name            log_message
       Parser              docker        
       Preserve_Key        On
       Reserve_Data        On 

在两个过滤器之前消息是:

{
  "time": "2022-05-10T19:43:04.655207298Z",
  "stream": "stdout",
  "_p": "F",
  "log": "{\"timestamp\":\"2022-05-10 19:43:04.654\",\"level\":\"DEBUG\",\"nested\":"{\"key1\":\"value\",\"key2\":\"value\"}",\"context\":\"default\"}",
  "log_processed": {
      "timestamp": "2022-05-10 19:43:04.654",
      "level": "DEBUG",     
      "message": "{\"key1\":\"value\",\"key2\":\"value\"}",
      "context": "default"
   }
 }

经过两次过滤后一级嵌套的json被解析为:

{
  "time": "2022-05-10T19:43:04.655207298Z",
  "stream": "stdout",
  "_p": "F",
  "log": "{\"timestamp\":\"2022-05-10 19:43:04.654\",\"level\":\"DEBUG\",\"nested\":"{\"key1\":\"value\",\"key2\":\"value\"}",\"context\":\"default\"}",
  "log_processed": {
      "timestamp": "2022-05-10 19:43:04.654",
      "level": "DEBUG",     
      "message": "{\"key1\":\"value\",\"key2\":\"value\"}",
      "context": "default"
   },
 log_message: "{\"key1\":\"value\",\"key2\":\"value\"}".
 "key1" : "value",
 "key2" : "value"
}

您可以尝试多次应用具有提升操作的嵌套插件。