apache-spark - Apache Spark Web UI 中的 "Stage Skipped"是什么意思？

Question

来自我的 Spark UI。跳过是什么意思？

Answer 1

通常这意味着数据已经从缓存中获取，不需要重新执行给定的阶段。它与您的 DAG 一致，表明下一阶段需要改组( reduceByKey )。每当涉及洗牌时 Spark automatically caches generated data :

Shuffle also generates a large number of intermediate files on disk. As of Spark 1.3, these files are preserved until the corresponding RDDs are no longer used and are garbage collected. This is done so the shuffle files don’t need to be re-created if the lineage is re-computed.