json - 从日志Logstash提取JSON

Question

我正在使用Logstach和logstach转发器提取我的日志文件，我有一些包含JSON格式的日志文件:

[2015-11-05 17:39:22.200] [INFO] dashboard - request :{ "user": "admin", "headers": {"host":"localhost:0000","connection":"keep-alive","accept":"application/json, text/plain, */*","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
"referer":"http://localhost:0000/","accept-encoding":"gzip, deflate, sdch","accept-language":"fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4","cookie":"splunkweb_csrf_token_8000=0000000000000000; _ga=GA1.1.0000000.1445436724; connect.sid=s%3AmlK46TZsFa202R5o3nwuHTbmHjehmJiO.JxlNGOXWuY%2Fp0RenTWsxLLDZvVgt8aVQ%2FpKryJsGCpw"},
"method": "GET", "url" : "/count_event", "params" :{"_type":"twitter,facebook,forum","year":"2013,2014,2015","month":"January,February,March,April,May,June,July,August,September,October,November,December"}}

因此，我需要在ES中保存以下信息:

时间戳:2015-11-05 17:39:22.200
类型:信息
msg:JSON中的所有信息

这是我的logstach文件配置:

input {
   lumberjack {
      port => 5043
      type => "logs"
      ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
      ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
   }
}
filter {
    grok {
      patterns_dir => ["/home/logstash/logstach-2.0.0/pattern"]
      add_tag => [ "valid" ]
      match => { "message" => "^\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{DATA:typemessage}\] %{DATA:appname} \- %{GREEDYDATA:msg}}
    }

   date {
     match => ["timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
     remove_field => ["timestamp"]
   }

    if "valid" not in [tags] {
      drop { }
    }
}
output {
   elasticsearch {
         hosts =>"192.168.1.153:9200"
         index =>"logs"
   }
   stdout { codec => rubydebug }
}

通过这种配置，我在味精字段上得到了一种字符串格式。

Answer 1

您需要告诉logstash将字段解析为json。 json{} filter用于此目的。在“msg”字段中输入。

但是请注意，您的原始代码应该在“msg”字段中包含输入的“request:”部分，该部分无效。您将需要调整grok模式，以仅将有效的json放入发送到json过滤器的字段中。