作者热门文章
- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
Spark 任务因“找不到 id 的搜索上下文”而失败。我尝试了几个选项,例如
spark.es.input.max.docs.per.partition 250
spark.es.scroll.size 100
spark.es.batch.size.bytes 32mb
但任务仍然失败。我们正在使用: 星火版本:2.3.1, Elasticsearch 集群版本:6.5.4, elasticsearch-spark版本:6.5.4
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: search_context_missing_exception: No search context found for id [4872076]
{"scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAABKV4wWRTJHQW9CdEZRTHVqMldEWnQxTUJnQQ=="}
at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:443)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:400)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:382)
at org.elasticsearch.hadoop.rest.RestClient.scroll(RestClient.java:458)
at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:323)
at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:115)
at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:533)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1092)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:347)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:298)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:42)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:336)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:300)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:112)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:384)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
如有任何帮助,我们将不胜感激。
最佳答案
增加以下参数的值对我有用。
es.scroll.keepalive=120m (default value is 10m)
引用:https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
关于scala - org.elasticsearch.hadoop.rest.EsHadoopRemoteException : search_context_missing_exception: No search context found for id 问题,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/54062991/
我是一名优秀的程序员,十分优秀!