apache-spark - Spark 流断言失败 : Failed to get records for spark-executor-a-group a-topic 7 244723248 after polling for 4096

Question

Kafka DirectStream 的 Spark Streaming 问题:

spark streaming assertion failed: Failed to get records for spark-executor-a-group a-topic 7 244723248 after polling for 4096

尝试:

1) 调整增加 spark.streaming.kafka.consumer.poll.ms

-- 从 512 到 4096，失败较少，但即使 10 秒失败仍然存在

2) 将执行器内存从 1G 调整为 2G

- 部分工作，更不用说失败了

3) https://issues.apache.org/jira/browse/SPARK-19275

-- 当流媒体持续时间都小于 8 秒时仍然失败(“session.timeout.ms”->“30000”)

4) 尝试 Spark 2.1

——问题依旧

使用 Scala 2.11.8，Kafka 版本:0.10.0.0，Spark 版本:2.0.2

Spark 配置

.config("spark.cores.max", "4")
.config("spark.default.parallelism", "2")
.config("spark.streaming.backpressure.enabled", "true")
.config("spark.streaming.receiver.maxRate", "1024")
.config("spark.streaming.kafka.maxRatePerPartition", "256")
.config("spark.streaming.kafka.consumer.poll.ms", "4096")
.config("spark.streaming.concurrentJobs", "2")

使用 spark-streaming-kafka-0-10-assembly_2.11-2.1.0.jar

错误堆栈:

at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:74)
at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:228)
at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:194)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.foreach(KafkaRDD.scala:194)
...
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:109)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:108)
at com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:142)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:108)
...
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

由于此故障，从 Kafka 丢失了 1% 以上的块数据:( 请帮忙!

Answer 1

当前解决方案:

增加 num.network.threads在 kafka/config/server.properties 中，默认为 3

增加 spark.streaming.kafka.consumer.poll.ms值~!一个大...
没有配置 spark.streaming.kafka.consumer.poll.ms，它使用的是 spark.network.timeout，它是 120 秒——导致一些问题

可选步骤:减少“max.poll.records”，默认为 500

可选步骤:使用 Future{} 并行运行时间成本任务