elasticsearch - 使用filebeat将结构化日志数据直接推送到elasticsearch

Question

我已将 filebeat 配置为收集我的结构化日志输出(绿地项目，因此每个日志条目都是预定义格式的 JSON 文档)并将其直接发布到 ELS。

示例日志文件摘录(请注意，additional 是自由形式，所有其他属性都是固定的。对于这篇文章来说格式很好，但每个顶级对象都在文件中的一行中):

{
    "TimeUtc": "2016-09-23T14:13:02.217520245Z",
    "ServiceKey": "MAAS_SVC",
    "Title": "Get All Campaigns - Start",
    "Additional": {
        "HTTPRequest": {
            "Method": "GET",
            "URL": {
                "Scheme": "",
                "Opaque": "",
                "User": null,
                "Host": "",
                "Path": "/admin/campaigns",
                "RawPath": "",
                "ForceQuery": false,
                "RawQuery": "",
                "Fragment": ""
            },
            "Proto": "HTTP/1.1",
            "ProtoMajor": 1,
            "ProtoMinor": 1,
            "Header": {
                "Accept": ["*/*"],
                "Accept-Encoding": ["gzip, deflate"],
                "Connection": ["keep-alive"],
                "Requestkey": ["78478050-47f0-4d0d-44e8-615d0599574a"],
                "User-Agent": ["python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"]
            },
            "Body": {
                "Closer": {
                    "Reader": null
                }
            },
            "ContentLength": 0,
            "TransferEncoding": null,
            "Close": false,
            "Host": "xxxxxxxxx",
            "Form": null,
            "PostForm": null,
            "MultipartForm": null,
            "Trailer": null,
            "RemoteAddr": "xxx.xxx.xxx.xxx",
            "RequestURI": "/admin/campaigns",
            "TLS": null,
            "Cancel": ,
            "Response": null
        }
    },
    "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
    "HostAddress": "xxxxxxxxx"
} 

这导致 filebeat 向 ELS 发出以下请求:

{
    "@timestamp": "2016-10-12T13:53:21.597Z",
    "beat": {
        "hostname": "7bca0e28e69e",
        "name": "7bca0e28e69e"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "message": "{\"TimeUtc\":\"2016-09-23T14:13:02.217520245Z\",\"ServiceKey\":\"MAAS_SVC\",\"Title\":\"Get All Campaigns - Start\",\"Additional\":{\"HTTPRequest\":{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/admin/campaigns\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Connection\":[\"keep-alive\"],\"Requestkey\":[\"78478050-47f0-4d0d-44e8-615d0599574a\"],\"User-Agent\":[\"python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic\"]},\"Body\":{\"Closer\":{\"Reader\":null}},\"ContentLength\":0,\"TransferEncoding\":null,\"Close\":false,\"Host\":\"bistromath.marathon.mesos:40072\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.1.70:42854\",\"RequestURI\":\"/admin/campaigns\",\"TLS\":null,\"Cancel\":,\"Response\":null}},\"RequestKey\":\"78478050-47f0-4d0d-44e8-615d0599574a\",\"HostAddress\":\"ba47316c9c45\"}",
    "offset": 0,
    "source": "/filebeat/log-harvest/maas-service-single.log",
    "type": "log"
}

我能否阻止 filebeat 转义我的日志 JSON，使其成为嵌套对象而不是字符串，或者我是否需要修补 filebeat？

Answer 1

可以在 Filebeat 5.x 中解析 JSON 消息，但不能在 Filebeat 1.x 中解析。 json选项可以在配置文件中指定。

如果您仅限于使用 Filebeat 1.x，那么您需要使用 Logstash 来解析来自 message 字段的 JSON 数据。您将配置 Filebeat -> Logstash -> Elasticsearch。

Filebeat 5.x 配置:

filebeat:
  prospectors:
    - paths:
        - input.json
      json.message_key: Title
      json.keys_under_root: true
      json.add_error_key: true

output:
  console:
    pretty: true

示例输出:

{
  "@timestamp": "2016-10-12T22:40:16.338Z",
  "Additional": {
    "HTTPRequest": {
      "Body": {
        "Closer": {}
      },
      "Close": false,
      "ContentLength": 0,
      "Header": {
        "Accept": [
          "*/*"
        ],
        "Accept-Encoding": [
          "gzip, deflate"
        ],
        "Connection": [
          "keep-alive"
        ],
        "Requestkey": [
          "78478050-47f0-4d0d-44e8-615d0599574a"
        ],
        "User-Agent": [
          "python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"
        ]
      },
      "Host": "xxxxxxxxx",
      "Method": "GET",
      "Proto": "HTTP/1.1",
      "ProtoMajor": 1,
      "ProtoMinor": 1,
      "RemoteAddr": "xxx.xxx.xxx.xxx",
      "RequestURI": "/admin/campaigns",
      "URL": {
        "ForceQuery": false,
        "Fragment": "",
        "Host": "",
        "Opaque": "",
        "Path": "/admin/campaigns",
        "RawPath": "",
        "RawQuery": "",
        "Scheme": ""
      }
    }
  },
  "HostAddress": "xxxxxxxxx",
  "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
  "ServiceKey": "MAAS_SVC",
  "TimeUtc": "2016-09-23T14:13:02.217520245Z",
  "Title": "Get All Campaigns - Start",
  "beat": {
    "hostname": "host",
    "name": "host"
  },
  "input_type": "log",
  "offset": 919,
  "source": "input.json",
  "type": "log"
}

注意:您发布的 JSON 数据无效。 Cancel 字段缺少一个值。在通过 Filebeat 运行数据之前，我将其设置为 null。