java - 无法执行多个 Spark 作业 "Initial job has not accepted any resources"

Question

使用独立 Spark Java 执行以下代码片段，我收到的状态始终为 WAITING，并出现以下错误。当我尝试添加 Print 语句时，它不起作用。是否有任何我可能错过的配置来运行多个作业？

15/09/18 15:02:56 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MapPartitionsRDD[2] at filter at SparkTest.java:143)

15/09/18 15:02:56 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

15/09/18 15:03:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/09/18 15:03:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/09/18 15:03:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

JavaRDD<String> words = input.flatMap(new FlatMapFunction<String, String>()    //Ln:143
        {
            public Iterable<String> call(String x)
            {
                return Arrays.asList(x.split(" "));
            }
        });
// Count all the words
System.out.println("Total words is" + words.count())

Answer 1

此错误消息意味着您的应用程序向集群请求的资源多于集群当前可以提供的资源，即多于集群中可用的内核或 RAM。

造成这种情况的原因之一可能是您已经有一个正在运行的作业耗尽了所有可用的核心。

发生这种情况时，您的作业很可能正在等待另一个作业完成并释放资源。

您可以在 Spark UI 中检查这一点。