elasticsearch - 如何为 ES 批量 API 形成 JSON

Question

我必须在 ES 中注入(inject) 1 亿条日志。我正在为此尝试 ES 批量 API。但我无法创建批量 API 所需的 JSON 文件格式。

示例 JSON 数据:

{"_index":"alias270_logs_auqaautoadsync3co-2019-07","_type":"elastica_logs","_id":"PT5PRwAqzaRpsQOz1lWHeT","_score":null,"_source":{"city":"Pune","Object_type":"Session","account_type":"Internal","severity":"informational","facility":"Box","country":"India","Resource_Id":"testuser1@securletbeatle.com","longitude":"73.8626","instance":"699178","host":"199.85.125.11","_latency":166,"Activity_type":"Login","location":"Pune (India)","__source":"API","created_timestamp":"2019-07-14T15:45:31","_domain":"securletbeatle.com","latitude":"18.5332","inserted_timestamp":"2019-07-14T15:45:31","user_name":"fiftythousand28455","message":"User logged in","user":"fiftythousand28455@elasticaqa.net","test_id":"NspjeLc8TQ8Q_deloitte"},"sort":[1562223465000]}

我以我的方式尝试并获得了这样的设置 JSON 文件

{ "index":{ "_index": "alias270_logs_auqaautoadsync3co-2019-07", "_type": "elastica_logs"} }

{"_source":{"city":"Pune","Object_type":"Session","account_type":"Internal","severity":"informational","facility":"Box","country":"India","Resource_Id":"testuser1@securletbeatle.com","longitude":"73.8626","instance":"699178","host":"199.85.125.11","_latency":166,"Activity_type":"Login","location":"Pune (India)","__source":"API","created_timestamp":"2019-07-14T15:45:31","_domain":"securletbeatle.com","latitude":"18.5332","inserted_timestamp":"2019-07-14T15:45:31","user_name":"fiftythousand28455","message":"User logged in","user":"fiftythousand28455@elasticaqa.net","test_id":"NspjeLc8TQ8Q_deloitte"} }

postman 上的错误消息:

“错误”:“MapperParsingException[解析失败，文档为空]”

Answer 1

REST API 端点是 /_bulk ，并且它期望以下换行符分隔 JSON (NDJSON) 结构体:

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n

注意:最后一行数据必须以换行符 \n 结尾.每个换行符前面都可以有一个回车符 \r .当向该端点发送请求时， Content-Type header 应设置为 application/x-ndjson .

在 Kibana 上，

POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }

使用 curl ，

curl -XPOST localhost:9200/index_local/my_doc_type/_bulk --data-binary  @/home/data.json

在哪里 /home/data.json应该是这样的:

{"index":{}}
{"Amount": "47280", "Quantity": "206", "Id": "975463711"}
{"index":{}}
{"Amount": "21505", "Quantity": "222", "Id": "975463943"}
{"index":{}}
{"Amount": "772107", "Quantity": "735", "Id": "974920111"}