使用 decode_json_fields 时出现 elasticsearch filebeat mapper_parsing_exception

Question

我有 ECK 设置，我正在使用 filebeat 将日志从 Kubernetes 发送到 elasticsearch。

我最近在我的配置中添加了 decode_json_fields 处理器，这样我就可以解码通常在 message 字段中的 json。

      - decode_json_fields:
          fields: ["message"]
          process_array: false
          max_depth: 10
          target: "log"
          overwrite_keys: true
          add_error_key: true

但是，自添加日志以来，日志已停止出现。

示例日志:

{
  "_index": "filebeat-7.9.1-2020.10.01-000001",
  "_type": "_doc",
  "_id": "wF9hB3UBtUOF3QRTBcts",
  "_score": 1,
  "_source": {
    "@timestamp": "2020-10-08T08:43:18.672Z",
    "kubernetes": {
      "labels": {
        "controller-uid": "9f3f9d08-cfd8-454d-954d-24464172fa37",
        "job-name": "stream-hatchet-cron-manual-rvd"
      },
      "container": {
        "name": "stream-hatchet-cron",
        "image": "<redacted>.dkr.ecr.us-east-2.amazonaws.com/stream-hatchet:v0.1.4"
      },
      "node": {
        "name": "ip-172-20-32-60.us-east-2.compute.internal"
      },
      "pod": {
        "uid": "041cb6d5-5da1-4efa-b8e9-d4120409af4b",
        "name": "stream-hatchet-cron-manual-rvd-bh96h"
      },
      "namespace": "default"
    },
    "ecs": {
      "version": "1.5.0"
    },
    "host": {
      "mac": [],
      "hostname": "ip-172-20-32-60",
      "architecture": "x86_64",
      "name": "ip-172-20-32-60",
      "os": {
        "codename": "Core",
        "platform": "centos",
        "version": "7 (Core)",
        "family": "redhat",
        "name": "CentOS Linux",
        "kernel": "4.9.0-11-amd64"
      },
      "containerized": false,
      "ip": []
    },
    "cloud": {
      "instance": {
        "id": "i-06c9d23210956ca5c"
      },
      "machine": {
        "type": "m5.large"
      },
      "region": "us-east-2",
      "availability_zone": "us-east-2a",
      "account": {
        "id": "<redacted>"
      },
      "image": {
        "id": "ami-09d3627b4a09f6c4c"
      },
      "provider": "aws"
    },
    "stream": "stdout",
    "message": "{\"message\":{\"log_type\":\"cron\",\"status\":\"start\"},\"level\":\"info\",\"timestamp\":\"2020-10-08T08:43:18.670Z\"}",
    "input": {
      "type": "container"
    },
    "log": {
      "offset": 348,
      "file": {
        "path": "/var/log/containers/stream-hatchet-cron-manual-rvd-bh96h_default_stream-hatchet-cron-73069980b418e2aa5e5dcfaf1a29839a6d57e697c5072fea4d6e279da0c4e6ba.log"
      }
    },
    "agent": {
      "type": "filebeat",
      "version": "7.9.1",
      "hostname": "ip-172-20-32-60",
      "ephemeral_id": "6b3ba0bd-af7f-4946-b9c5-74f0f3e526b1",
      "id": "0f7fff14-6b51-45fc-8f41-34bd04dc0bce",
      "name": "ip-172-20-32-60"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-10-08T08:43:18.672Z"
    ],
    "suricata.eve.timestamp": [
      "2020-10-08T08:43:18.672Z"
    ]
  }
}

在 filebeat 日志中，我可以看到以下错误:

2020-10-08T09:25:43.562Z WARN [elasticsearch] elasticsearch/client.go:407 Cannotindex eventpublisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x36b243a0,ext:63737745936, loc:(*time.Location)(nil)}, Meta:null,Fields:{"agent":{"ephemeral_id":"5f8afdba-39c3-4fb7-9502-be7ef8f2d982","hostname":"ip-172-20-32-60","id":"0f7fff14-6b51-45fc-8f41-34bd04dc0bce","name":"ip-172-20-32-60","type":"filebeat","version":"7.9.1"},"cloud":{"account":{"id":"700849607999"},"availability_zone":"us-east-2a","image":{"id":"ami-09d3627b4a09f6c4c"},"instance":{"id":"i-06c9d23210956ca5c"},"machine":{"type":"m5.large"},"provider":"aws","region":"us-east-2"},"ecs":{"version":"1.5.0"},"host":{"architecture":"x86_64","containerized":false,"hostname":"ip-172-20-32-60","ip":["172.20.32.60","fe80::af:9fff:febe:dc4","172.17.0.1","100.96.1.1","fe80::6010:94ff:fe17:fbae","fe80::d869:14ff:feb0:81b3","fe80::e4f3:b9ff:fed8:e266","fe80::1c19:bcff:feb3:ce95","fe80::fc68:21ff:fe08:7f24","fe80::1cc2:daff:fe84:2a5a","fe80::3426:78ff:fe22:269a","fe80::b871:52ff:fe15:10ab","fe80::54ff:cbff:fec0:f0f","fe80::cca6:42ff:fe82:53fd","fe80::bc85:e2ff:fe5f:a60d","fe80::e05e:b2ff:fe4d:a9a0","fe80::43a:dcff:fe6a:2307","fe80::581b:20ff:fe5f:b060","fe80::4056:29ff:fe07:edf5","fe80::c8a0:5aff:febd:a1a3","fe80::74e3:feff:fe45:d9d4","fe80::9c91:5cff:fee2:c0b9"],"mac":["02:af:9f:be:0d:c4","02:42:1b:56:ee:d3","62:10:94:17:fb:ae","da:69:14:b0:81:b3","e6:f3:b9:d8:e2:66","1e:19:bc:b3:ce:95","fe:68:21:08:7f:24","1e:c2:da:84:2a:5a","36:26:78:22:26:9a","ba:71:52:15:10:ab","56:ff:cb:c0:0f:0f","ce:a6:42:82:53:fd","be:85:e2:5f:a6:0d","e2:5e:b2:4d:a9:a0","06:3a:dc:6a:23:07","5a:1b:20:5f:b0:60","42:56:29:07:ed:f5","ca:a0:5a:bd:a1:a3","76:e3:fe:45:d9:d4","9e:91:5c:e2:c0:b9"],"name":"ip-172-20-32-60","os":{"codename":"Core","family":"redhat","kernel":"4.9.0-11-amd64","name":"CentOSLinux","platform":"centos","version":"7(Core)"}},"input":{"type":"container"},"kubernetes":{"container":{"image":"700849607999.dkr.ecr.us-east-2.amazonaws.com/stream-hatchet:v0.1.4","name":"stream-hatchet-cron"},"labels":{"controller-uid":"a79daeac-b159-4ba7-8cb0-48afbfc0711a","job-name":"stream-hatchet-cron-manual-c5r"},"namespace":"default","node":{"name":"ip-172-20-32-60.us-east-2.compute.internal"},"pod":{"name":"stream-hatchet-cron-manual-c5r-7cx5d","uid":"3251cc33-48a9-42b1-9359-9f6e345f75b6"}},"log":{"level":"info","message":{"log_type":"cron","status":"start"},"timestamp":"2020-10-08T09:25:36.916Z"},"message":"{"message":{"log_type":"cron","status":"start"},"level":"info","timestamp":"2020-10-08T09:25:36.916Z"}","stream":"stdout"},Private:file.State{Id:"native::30998361-66306", PrevId:"",Finished:false, Fileinfo:(*os.fileStat)(0xc001c14dd0),Source:"/var/log/containers/stream-hatchet-cron-manual-c5r-7cx5d_default_stream-hatchet-cron-4278d956fff8641048efeaec23b383b41f2662773602c3a7daffe7c30f62fe5a.log",Offset:539, Timestamp:time.Time{wall:0xbfd7d4a1e556bd72,ext:916563812286, loc:(*time.Location)(0x607c540)}, TTL:-1,Type:"container", Meta:map[string]string(nil),FileStateOS:file.StateOS{Inode:0x1d8ff59, Device:0x10302},IdentifierName:"native"}, TimeSeries:false}, Flags:0x1,Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400):{"type":"mapper_parsing_exception","reason":"failed to parse field[log.message] of type [keyword] in document with id'56aHB3UBLgYb8gz801DI'. Preview of field's value: '{log_type=cron,status=start}'","caused_by":{"type":"illegal_state_exception","reason":"Can'tget text on a START_OBJECT at 1:113"}}

它会抛出一个错误，因为显然 log.message 是“关键字”类型，但这在索引映射中不存在。

我认为这可能是 "target": "log" 的问题，所以我尝试将其更改为任意值，如 "my_parsed_message"或 "m_log"或 "mlog"，我得到了相同的结果所有这些都是错误的。

{"type":"mapper_parsing_exception","reason":"failed to parse field[mlog.message] of type [keyword] in document with id'J5KlDHUB_yo5bfXcn2LE'. Preview of field's value: '{log_type=cron,status=end}'","caused_by":{"type":"illegal_state_exception","reason":"Can'tget text on a START_OBJECT at 1:217"}}

弹性版本:7.9.2

Answer 1

问题是您的一些 JSON 消息包含一个 message 字段，该字段有时是一个简单的字符串，有时是一个嵌套的 JSON 对象(就像您在问题中显示的情况一样)。

创建此索引后，解析的第一条消息可能是字符串，因此已修改映射以添加以下字段 (line 10553):

"mlog": {
   "properties": {
       ...
       "message": {
          "type": "keyword",
          "ignore_above": 1024
       },
   }
}

您会发现 my_parsed_message ( line 10902 )、my_parsed_logs ( line 10742 ) 等的相同模式...

因此 message 的下一条消息是一个 JSON 对象，比如

{"message":{"log_type":"cron","status":"start"}, ...

不会工作，因为它是一个对象，而不是字符串...

查看自定义 JSON 的字段，您似乎无法真正控制它们的分类法(即命名)或它们包含的内容...

如果你真的愿意在这些自定义字段中搜索(我认为你是因为你正在解析字段，否则你只是存储字符串化的 JSON)，那么我只能建议开始弄清楚适当的分类法，以确保它们都获得标准类型。

如果您只关心记录数据，那么我建议简单地使用 disable the indexing该消息字段。另一种解决方案是设置 dynamic: false在您的映射中忽略这些字段，即不修改您的映射。