api - Spark 作业已提交 - 正在等待(TaskSchedulerImpl : Initial job not accepted)

Question

调用 API 来提交作业。响应状态 - 正在运行

在集群 UI 上 -

Worker (slave) - worker-20160712083825-172.31.17.189-59433 is Alive

Core 1 out of 2 used

Memory 1Gb out of 6 used

运行应用程序

app-20160713130056-0020 - Waiting since 5hrs

Cores - unlimited

应用程序的工作描述

活跃阶段

reduceByKey at /root/wordcount.py:23

待定阶段

takeOrdered at /root/wordcount.py:26

运行驱动程序 -

stderr log page for driver-20160713130051-0025 

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

根据Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources从站尚未启动 - 因此它没有资源。

但是就我而言 - Slave 1 正在工作

根据Unable to Execute More than a spark Job "Initial job has not accepted any resources"我正在使用部署模式=集群(不是客户端)，因为我有1个主站1个从站，并且提交API正在通过Postman/任何地方调用

集群还有可用的核心、RAM、内存 - Still Job 会抛出错误由 UI 传达

根据TaskSchedulerImpl: Initial job has not accepted any resources;我分配了

~/spark-1.5.0/conf/spark-env.sh

Spark环境变量

SPARK_WORKER_INSTANCES=1
SPARK_WORKER_MEMORY=1000m
SPARK_WORKER_CORES=2

在从站中复制这些

sudo /root/spark-ec2/copy-dir /root/spark/conf/spark-env.sh

上述问题答案中的所有情况均适用，但仍没有找到解决方案。因此，因为我正在使用 API 和 Apache SPark - 也许需要一些其他帮助。

Edited July 18,2016

Wordcount.py - My PySpark application code -

from pyspark import SparkContext, SparkConf

logFile = "/user/root/In/a.txt"

conf = (SparkConf().set("num-executors", "1"))

sc = SparkContext(master = "spark://ec2-54-209-108-127.compute-1.amazonaws.com:7077", appName = "MyApp", conf = conf)
print("in here")
lines = sc.textFile(logFile)
print("text read")
c = lines.count()
print("lines counted")

错误

Starting job: count at /root/wordcount.py:11
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Got job 0 (count at /root/wordcount.py:11) with 2 output partitions
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at /root/wordcount.py:11)
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (PythonRDD[2] at count at /root/wordcount.py:11), which has no missing parents
16/07/18 07:46:39 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.6 KB, free 56.2 KB)
16/07/18 07:46:39 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.4 KB, free 59.7 KB)
16/07/18 07:46:39 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.31.17.189:43684 (size: 3.4 KB, free: 511.5 MB)
16/07/18 07:46:39 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
16/07/18 07:46:39 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (PythonRDD[2] at count at /root/wordcount.py:11)
16/07/18 07:46:39 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/07/18 07:46:54 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

根据Spark UI showing 0 cores even when setting cores in App ,

Spark WebUI 声明使用了零个核心并且无限期等待没有任务运行。该应用程序在运行时或核心期间也不使用任何内存，并且在启动时立即进入等待状态

Spark 版本 1.6.1乌类图亚马逊EC2

Answer 1

我也有同样的问题。以下是我在发生这种情况时的评论。

1:17:46 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

我注意到它只发生在 scala shell 的第一次查询期间，我在其中运行从 hdfs 获取数据的操作。

出现问题时，WebUI 会显示没有任何正在运行的应用程序。

URL: spark://spark1:7077
REST URL: spark://spark1:6066 (cluster mode)
Alive Workers: 4
Cores in use: 26 Total, 26 Used
Memory in use: 52.7 GB Total, 4.0 GB Used
Applications: 0 Running, 0 Completed
Drivers: 0 Running, 0 Completed 
Status: ALIVE

好像有什么东西无法启动，我无法确切地分辨出是哪一个。

但是，第二次重新启动集群会将应用程序值设置为 1一切正常。

URL: spark://spark1:7077
REST URL: spark://spark1:6066 (cluster mode)
Alive Workers: 4
Cores in use: 26 Total, 26 Used
Memory in use: 52.7 GB Total, 4.0 GB Used
Applications: 1 Running, 0 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE

我仍在调查，这种快速解决方法可以节省最终解决方案的时间。