json - 如何在 Log-stash HTTP 输出中映射嵌套的 JSON

Question

我正在使用 Logstash 将 JSON 消息输出到 API。我正在使用“映射”属性来映射我的消息。看，下面是我的托运人配置。

output {
    stdout { }
     http {
        url => "http://localhost:8087/messages"
        http_method => "post"
        format => "json"
        mapping => ["MessageId","654656","TimeStamp","2001-12-31T12:00:00","CorrelationId","986565","MessageType","%{log_MessageType}" ,"MessageTitle","%{log_MessageTitle}","Message","%{log_Message}"]
    }
}

此配置工作正常，并产生以下输出:

{
  "MessageId": "654656",
  "TimeStamp": "2001-12-31T12:00:00",
  "CorrelationId": "986565",
  "MessageType": "INFO",
  "MessageTitle": "TestTittle",
  "Message": "Sample Message"
}

输入日志条目:

TID: [0] [ESB] [2016-05-30 23:02:02,602]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Configured Registry in 572ms {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService}

Grok 模式:

TID:%{SPACE}\[%{INT:log_SourceSystemId}\]%{SPACE}\[%{DATA:log_ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:log_TimeStamp}\]%{SPACE}%{LOGLEVEL:log_MessageType}%{SPACE}{%{JAVACLASS:log_MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_Message}

问题陈述:

我想通过 HTTP 映射来跟踪输出。我想要一个嵌套的 JSON 类型在我的消息中，我应该如何在映射标签中添加它。

预期输出:

{
  "MessageId": "654656",
  "TimeStamp": "2001-12-31T12:00:00",
  "CorrelationId": "986565",
  "MessageType": "INFO",
  "MessageTitle": "TestTittle",
  "Message": "Sample Message",
  "MessageDetail": {
    "FieldA": "65656",
    "FieldB": "192.168.1.1",
    "FieldC": "sample value"

  }
}

我尝试了几个选项，但我收到错误。

Answer 1

使用 message 无法做到这一点http 中的映射输出。该映射只能创建单级 JSON。

但是，您可以做的是在到达 http 之前构建您的 JSON 消息。使用 mutate/add_field 输出筛选。

filter {
   grok {
       match => { "message" => "TID:%{SPACE}\[%{INT:SourceSystemId}\]%{SPACE}\[%{DATA:ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:log_TimeStamp}\]%{SPACE}%{LOGLEVEL:log_MessageType}%{SPACE}{%{JAVACLASS:log_MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_Message}" }
   }

   # add additional fields in your event here
   mutate {
      gsub => [
        "log_TimeStamp", "\s", "T",
        "log_TimeStamp", ",", "."
      ]
      add_field => {
        "MessageId" => "654656"
        "TimeStamp" => "%{log_TimeStamp}"
        "CorrelationId" => "986565"
        "MessageType" => "%{log_MessageType}"
        "MessageTitle" => "%{log_MessageTitle}"
        "Message" => "%{log_Message}"
        "[MessageDetail][FieldA]" => "65656"
        "[MessageDetail][FieldB]" => "192.168.1.1"
        "[MessageDetail][FieldC]" => "sample value"
      }
      remove_field => ["@version", "@timestamp", "host", "message", "SourceSystemId", "ProcessName", "log_TimeStamp", "log_MessageType", "log_MessageTitle", "log_Message"]
   }
}
output {
   stdout { codec => "rubydebug" }
   http {
      url => "http://localhost:8087/messages"
      http_method => "post"
      format => "json"
   }
}

您将获得您期望发布到 HTTP 端点的 JSON

{
         "MessageId": "654656",
         "TimeStamp": "2016-05-30T23:02:02.602",
     "CorrelationId": "986565",
       "MessageType": "INFO",
      "MessageTitle": "org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService",
           "Message": "Configured Registry in 572ms {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService}",
     "MessageDetail": {
        "FieldA": "65656"
        "FieldB": "192.168.1.1"
        "FieldC": "sample value"
     }
}