apache-spark - 没有阶段正在运行，但 numRunningTasks != 0

Question

我的任务完成了，我得到了预期的 RDD 计数结果。我正在运行一个交互式 PySpark shell。我试图理解这个警告是什么意思:

WARN ExecutorAllocationManager: No stages are running, but numRunningTasks != 0

来自 Spark 内部 code我找到了这个:

    // If this is the last stage with pending tasks, mark the scheduler queue as empty
    // This is needed in case the stage is aborted for any reason
    if (stageIdToNumTasks.isEmpty) {
      allocationManager.onSchedulerQueueEmpty()
      if (numRunningTasks != 0) {
        logWarning("No stages are running, but numRunningTasks != 0")
        numRunningTasks = 0
      }
    }

有人可以解释一下吗？

我说的是 ID 为 0 的任务。



我可以通过 KMeans() 报告使用 Spark 的 MLlib 遇到这种行为，其中 the one of the two samples据说用更少的任务完成。我不确定这项工作是否会失败。

2  takeSample at KMeans.scala:355 2016/08/27 21:39:04   7 s 1/1 9600/9600
1  takeSample at KMeans.scala:355 2016/08/27 21:38:57   6 s 1/1 6608/9600

输入集是 100m 点，256 个维度。

PySpark的一些参数:master是yarn，mode是cluster，

spark.dynamicAllocation.enabled             false
# Better serializer - https://spark.apache.org/docs/latest/tuning.html#data-serialization
spark.serializer                            org.apache.spark.serializer.KryoSerializer
spark.kryoserializer.buffer.max             2000m

# Bigger PermGen space, use 4 byte pointers (since we have < 32GB of memory)
spark.executor.extraJavaOptions             -XX:MaxPermSize=512m -XX:+UseCompressedOops

# More memory overhead
spark.yarn.executor.memoryOverhead          4096
spark.yarn.driver.memoryOverhead            8192

spark.executor.cores                        8
spark.executor.memory                       8G

spark.driver.cores                          8
spark.driver.memory                         8G
spark.driver.maxResultSize                  4G

Answer 1

我们得到的是这个代码:

    ...
    // If this is the last stage with pending tasks, mark the scheduler queue as empty
    // This is needed in case the stage is aborted for any reason
    if (stageIdToNumTasks.isEmpty) {
      allocationManager.onSchedulerQueueEmpty()
      if (numRunningTasks != 0) {
        logWarning("No stages are running, but numRunningTasks != 0")
        numRunningTasks = 0
      }
    }
  }
}

来自 Spark 的 GitHub ，评论是迄今为止最好的解释。