json - [layers] 中 START_OBJECT 的未知键

Question

尝试将我的 json 格式的 tcpdump 发送到 elasticsearch 时，我收到以下错误:

curl -X PUT --data-binary @myjson 'localhost:9200/_bulk?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Unknown key for a START_OBJECT in [layers].",
        "line" : 1,
        "col" : 95
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Unknown key for a START_OBJECT in [layers].",
    "line" : 1,
    "col" : 95
  },
  "status" : 400
}

json 文件是使用带有“-T json”选项的 tshark 获得的。
json 文件是使用带有过滤器“{index: .[]}”和选项 -c 的 jq 修改的，因为 elasticsearch 需要一个条目以适合单行。
我正在使用标准配置的 elasticsearch 5.5.1。
jsonformatter 将 json 对象标记为有效。

产生错误的 json 对象如下所示:

{"index":{"_index":"packets-2017-08-04","_type":"pcap_file","_score":null,"_source":{"layers":{"frame":{"frame.encap_type":"25","frame.time":"Aug  5, 2001 13:10:06.559762000 CEST","frame.offset_shift":"0.000000000","frame.time_epoch":"1501773006.559765000","frame.time_delta":"0.000000000","frame.time_delta_displayed":"0.000000000","frame.time_relative":"0.000000000","frame.number":"1","frame.len":"200","frame.cap_len":"200","frame.marked":"0","frame.ignored":"0","frame.protocols":"sll:ethertype:ip:tcp:data"},"sll":{"sll.pkttype":"4","sll.hatype":"65135","sll.halen":"0","sll.etype":"0x00000800"},"ip":{"ip.version":"4","ip.hdr_len":"20","ip.dsfield":"0x00000010","ip.dsfield_tree":{"ip.dsfield.dscp":"4","ip.dsfield.ecn":"0"},"ip.len":"184","ip.id":"0x000093f2","ip.flags":"0x00000002","ip.flags_tree":{"ip.flags.rb":"0","ip.flags.df":"1","ip.flags.mf":"0"},"ip.frag_offset":"0","ip.ttl":"64","ip.proto":"6","ip.checksum":"0x0000ef4b","ip.checksum.status":"2","ip.src":"0.0.00","ip.addr":"0.0.0.0","ip.src_host":"0.0.0.0","ip.host":"0.0.0.0","ip.dst":"0.0.0.0","ip.dst_host":"0.0.0.0","Source GeoIP: Germany":{"ip.geoip.src_country":"Germany","ip.geoip.country":"Germany","ip.geoip.src_city":"Frankfurt, 1","ip.geoip.city":"Berlin, 1","ip.geoip.src_asnum":"123","ip.geoip.asnum":"123","ip.geoip.src_lat":"701","ip.geoip.lat":"523,01","ip.geoip.src_lon":"2313,4","ip.geoip.lon":"12,13"},"Destination GeoIP: Germany":{"ip.geoip.dst_country":"Germany","ip.geoip.country":"Germany","ip.geoip.dst_asnum":"123","ip.geoip.asnum":"123","ip.geoip.dst_lat":"3321","ip.geoip.lat":"41","ip.geoip.dst_lon":"1","ip.geoip.lon":"2"}},"tcp":{"tcp.srcport":"41","tcp.dstport":"124","tcp.port":"234","tcp.stream":"3","tcp.len":"134","tcp.seq":"1","tcp.nxtseq":"133","tcp.ack":"4","tcp.hdr_len":"32","tcp.flags":"0x00000018","tcp.flags_tree":{"tcp.flags.res":"0","tcp.flags.ns":"0","tcp.flags.cwr":"0","tcp.flags.ecn":"0","tcp.flags.urg":"0","tcp.flags.ack":"1","tcp.flags.push":"1","tcp.flags.reset":"0","tcp.flags.syn":"0","tcp.flags.fin":"0","tcp.flags.str":"·······AP···"},"tcp.window_size_value":"223","tcp.window_size":"31","tcp.window_size_scalefactor":"-1","tcp.checksum":"0x0000b79c","tcp.checksum.status":"1","tcp.urgent_pointer":"0","tcp.options":"123","tcp.options_tree":{"No-Operation (NOP)":{"tcp.options.type":"1","tcp.options.type_tree":{"tcp.options.type.copy":"0","tcp.options.type.class":"0","tcp.options.type.number":"1"}},"Timestamps: TSval 1875055084, TSecr 5726840":{"tcp.option_kind":"8","tcp.option_len":"10","tcp.options.timestamp.tsval":"185084","tcp.options.timestamp.tsecr":"1116840"}},"tcp.analysis":{"tcp.analysis.bytes_in_flight":"123","tcp.analysis.push_bytes_sent":"133"}},"data":{"data.data":"01:01:02","data.len":"265"}}}}}

我的问题是:这个 json 有什么问题，以至于 elasticsearch 拒绝它？

Answer 1

这不是真正的 jq 问题 - START_OBJECT 的未知键是 Elasticsearch 错误。 [layers] 暗示问题出在那里的对象中，不幸的是在问题描述中被省略了，所以这里真的没什么可做的。

由于您指定的 jq 过滤器只是 {index:.[]} ， jq 对 json elasticsearch 提示的部分没有做任何事情。如果您的工作流程期望 jq 以某种方式更正该部分，则您需要更仔细地调查数据并使用更复杂的过滤器。

作为引用，elasticsearch test suite包含此特定错误的示例:

---
"junk in source fails":
  - do:
      catch: /Unknown key for a START_OBJECT in \[junk\]./
      reindex:
        body:
          source:
            junk: {}

希望这可以帮助。