elasticsearch - 如何从 Elasticsearch 获取最大_id

Question

我创建了一条河流，该河流每小时运行一次以从DB中获取数据(使用jdbc river插件)。

select * from orders

我不想选择所有记录，而是选择基于主键追加的数据。查询将是:

select * from orders where deviceid > '(Max Id in Elastic search)'

如何从 flex 搜索中获取最大_id？

Answer 1

由于ES坚持将"_id"值转换为字符串，因此似乎没有使用"_id"字段直接执行此操作的方法。但是有一种解决方法。

首先，我用几个文档设置了一个简单的索引，如下所示:

PUT /test_index
{
   "settings": {
      "number_of_shards": 1
   }
}

POST /test_index/_bulk
{"index":{"_index":"test_index","_type":"doc","_id":1}}
{"title":"first doc"}
{"index":{"_index":"test_index","_type":"doc","_id":2}}
{"title":"second doc"}
{"index":{"_index":"test_index","_type":"doc","_id":3}}
{"title":"third doc"}

然后我尝试使用 max aggregation，但出现错误，因为 "_id"是字符串:

POST /test_index/_search?search_type=count
{
   "aggs": {
      "max_id": {
         "max": {
            "field": "_id"
         }
      }
   }
}
...
{
   "error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[bQS7TqO9SfKSPQZYVXQBag][test_index][0]: ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]}]",
   "status": 500
}

所以那行不通。但是使用 "path" field中的 "_id"参数进行了一些修改。

所以我将索引重新定义为

DELETE /test_index

PUT /test_index
{
   "settings": {
      "number_of_shards": 1
   },
   "mappings": {
      "doc": {
         "_id": {
            "path": "doc_id"
         }
      }
   }
}

然后使用 "doc_id"路径为文档建立索引:

POST /test_index/_bulk
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"first doc","doc_id":1}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"second doc","doc_id":2}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"third doc","doc_id":3}

现在，如果我进行搜索，可以看到 "_id"仍然是字符串，但是 "doc_id"是整数:

POST /test_index/_search
...
{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 3,
      "max_score": 1,
      "hits": [
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "1",
            "_score": 1,
            "_source": {
               "title": "first doc",
               "doc_id": 1
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "2",
            "_score": 1,
            "_source": {
               "title": "second doc",
               "doc_id": 2
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "3",
            "_score": 1,
            "_source": {
               "title": "third doc",
               "doc_id": 3
            }
         }
      ]
   }
}

所以现在我可以轻松地使用最大聚合来找到最大id值:

POST /test_index/_search?search_type=count
{
   "aggs": {
      "max_id": {
         "max": {
            "field": "doc_id"
         }
      }
   }
}
...
{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 3,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "max_id": {
         "value": 3
      }
   }
}