database - 无法使用ElasticDump从映射导入_timestamp

Question

我有一些来自ElasticSearch数据库的JSON文件，我正尝试用ElasticDump导入它们。

这是映射文件:“mylog.mapping.json”

[
"{\"mylog\":{\"mappings\":{\"search_log\":{\"_timestamp\":{\"enabled\":true,\"store\":true},\"properties\":{\"preArray\":{\"type\":\"long\"},\"preId\":{\"type\":\"string\"},\"filteredSearch\":{\"type\":\"string\"},\"hits\":{\"type\":\"long\"},\"search\":{\"type\":\"string\"},\"searchType\":{\"properties\":{\"name\":{\"type\":\"string\"}}}}}}}}"
]

包含数据本身的文件:“mylog.json”

{"_index":"mylog","_type":"search_log","_id":"AU5AcRy7dbXLQfUndnNS","_score":1,"_source":{"searchType":{"name":"TypeSearchOne"},"search":"test","filteredSearch":"test","hits":1470,"preId":"","preArray":[47752,51493,52206,50159,52182,53243,43237,51329,42772,44938,44945,44952,42773,58319,43238,48963,52856,52185,47751,61542,51327,42028,51341,45356,44853,44939,48587,42774,43063,98779,46235,53533,47745,48844,44979,53209,47738,98781,47757,44948,44950,48832,97529,52186,96033,53002,48419,44943,44955,52179]},"fields":{"_timestamp":1435600231611}}
{"_index":"mylog","_type":"search_log","_id":"AU5AcSdcdbXLQfUndnNd","_score":1,"_source":{"searchType":{"name":"TypeSearchTwo"},"search":"squared","filteredSearch":"squared","hits":34,"preId":null,"preArray":null},"fields":{"_timestamp":1435600234333}}
{"_index":"mylog","_type":"search_log","_id":"AU5AcSiZdbXLQfUndnNj","_score":1,"_source":{"searchType":{"name":"TypeSearchOne"},"search":"test","filteredSearch":"test","hits":1354,"preId":"","preArray":[55808,53545,53543,53651,55937,53544,54943,54942,54941]},"fields":{"_timestamp":1435600234649}}

...

{"_index":"mylog","_type":"search_log","_id":"AU5DSVzLdbXLQfUndnPp","_score":1,"_source":{"searchType":{"name":"TypeSearchOne"},"search":"lee","filteredSearch":"lee","hits":39,"preId":"53133","preArray":null},"fields":{"_timestamp":1435647958219}}
{"_index":"mylog","_type":"search_log","_id":"AU5D7M42dbXLQfUndnR9","_score":1,"_source":{"searchType":{"name":"TypeSearchOne"},"search":"leerwww","filteredSearch":"leerwww","hits":39,"preId":"53133","preArray":null},"fields":{"_timestamp":1435658669622}}

在尝试将这些数据导入我的ElasticSearch服务器时，我尝试了以下ElasticDump命令:

elasticdump --input=/home/user/Desktop/LOGDATA/mylog.mapping.json --output=http://localhost:9200/mylog --type=mapping
elasticdump --input=/home/user/Desktop/LOGDATA/mylog.json --output=http://localhost:9200/mylog --type=data

此后，数据可用，但是 _timestamp字段无处可见。如果我检查映射，则将获得以下信息:

user@computer:~$ curl -XGET 'localhost:9200/mylog/_mapping'

{
    "mylog":{
        "mappings":{
            "search_log":{
                "properties":{
                    "preArray":{"type":"long"},
                    "preId":{"type":"string"},
                    "filteredSearch":{"type":"string"},
                    "hits":{"type":"long"},
                    "search":{"type":"string"},
                    "searchType":{"properties":{"name":{"type":"string"}}}
                }
            }
        }
    }
}

如您所见，即使在映射中指定了 _timestamp字段，该字段也不存在。为什么会发生这种情况？如何在不丢失时间戳的情况下导入数据？

Answer 1

从2.0开始， _timestamp is deprecated and a special type of field known as a meta-field。它仍然存在于5.0中(至少现在是这样)，但是您不应该依赖它，并且应该期望将其删除。

与其他元字段一样，您不应该能够修改其映射(例如，指定stored: true)，也不能将其设置为文档的一部分。

您应该做的是将字段设置为请求参数:

PUT my_index/my_type/1?timestamp=1435600231611
{"searchType":{"name":"TypeSearchOne"},"search":"test","filteredSearch":"test","hits":1470,"preId":"","preArray":[47752,51493,52206,50159,52182,53243,43237,51329,42772,44938,44945,44952,42773,58319,43238,48963,52856,52185,47751,61542,51327,42028,51341,45356,44853,44939,48587,42774,43063,98779,46235,53533,47745,48844,44979,53209,47738,98781,47757,44948,44950,48832,97529,52186,96033,53002,48419,44943,44955,52179]}

我对ElasticDump的了解不足，无法知道是否可以指示它做“正确的事情”，但实际上这里有一个更好的选择:

修改您的JSON输入以删除 _timestamp并将其替换为名为 timestamp(或您选择的任何名称)的普通字段。

"mappings": {
  "my_type": {
    "properties": {
      "timestamp": {
        "type": "date"
      },
      ...
    }
  }
}

请注意，您的ElasticDump输入将 _timestamp中的 fields而不是 source分隔开，因此您必须确保进行查找/替换以将它们正确地结合在一起:

},"fields":{"_timestamp"

应该:

,"timestamp"