apache-spark - “Stage Skipped”在Apache Spark Web UI中是什么意思？

Question

从我的Spark UI。跳过是什么意思？

Answer 1

通常，这意味着已经从缓存中获取了数据，并且无需重新执行给定的阶段。它与您的DAG一致，它表明下一阶段需要改组(reduceByKey)。每当涉及改组时，Spark automatically caches generated data:

Shuffle also generates a large number of intermediate files on disk. As of Spark 1.3, these files are preserved until the corresponding RDDs are no longer used and are garbage collected. This is done so the shuffle files don’t need to be re-created if the lineage is re-computed.