apache-spark - Spark 调度器与 Spark 堆栈中的独立调度器

Question

Spark核心中的调度器与以下Spark Stack(来自Learning Spark:Lightning-Fast Big Data Analysis一书)中的Standalone Scheduler有什么区别？

Answer 1

the overview of spark scheduling 中突出显示了这两者之间的区别。 :

Spark has several facilities for scheduling resources between computations. First, recall that, as described in the cluster mode overview, each Spark application (instance of SparkContext) runs an independent set of executor processes. The cluster managers that Spark runs on provide facilities for scheduling across applications. Second, within each Spark application, multiple “jobs” (Spark actions) may be running concurrently if they were submitted by different threads. This is common if your application is serving requests over the network. Spark includes a fair scheduler to schedule resources within each SparkContext.

简而言之:一方面，您所说的独立调度程序处于集群管理器级别，并处理多个 Spark 应用程序的资源。另一方面，应用程序中的多个作业由您所说的 Spark 核心调度程序管理。