apache-spark - 当执行人丢失时会发生什么？

Question

我收到这些消息:

16/05/22 13:33:53 ERROR YarnScheduler: Lost executor 61 on <host>: Executor heartbeat timed out after 134828 ms
16/05/22 13:33:53 WARN TaskSetManager: Lost task 25.0 in stage 12.0 (TID 2214, <host>): ExecutorLostFailure (executor 61 lost)

会产生替代执行器吗？

Answer 1

Will a replacement executor be spawned?

是的，会的。 Sparks DAGScheduler 及其较低级别的集群管理器实现(独立、YARN 或 Mesos)将注意到任务失败，并将负责重新安排该任务，作为执行的总体阶段的一部分。

DAGScheduler does three things in Spark (thorough explanations follow):

• Computes an execution DAG, i.e. DAG of stages, for a job.
• Determines the preferred locations to run each task on.
• Handles failures due to shuffle output files being lost.

有关更多信息，您可以在 Advanced Spark Tutorial 中找到并在 Mastering Apache Spark .