apache-spark - Kubernetes 上的 Spark 执行 - 驱动程序 pod 失败

Question

我尝试使用 spark 2.3 原生 kubernetes 部署功能在 kubernetes 集群上运行简单的 spark 代码。

我有一个 Kubernetes 集群正在运行。此时，spark 代码不会读取或写入数据。它从列表中创建一个 RDD 并打印出结果，只是为了验证在 spark 上运行 kubernetes 的能力。此外，还将 spark 应用程序 jar 复制到 kubernetes 容器镜像中。

下面是我运行的命令。
bin/spark-submit --master k8s://https://k8-master --deploy-mode cluster --name sparkapp --class com.sparrkonk8.rdd.MockWordCount --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=myapp/sparkapp:1.0.0 local:///SparkApp.jar

2018-03-06 10:31:28 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state: pod name: sparkapp-6e475a6ae18d3b7a89ca2b5f6ae7aae4-driver namespace: default labels: spark-app-selector -> spark-9649dd66e9a946d989e2136d342ef249, spark-role -> driver pod uid: 6d3e98cf-2153-11e8-85af-1204f474c8d2 creation time: 2018-03-06T15:31:23Z service account name: default volumes: default-token-vwxvr node name: 192-168-1-1.myapp.engg.com start time: 2018-03-06T15:31:23Z container images: dockerhub.com/myapp/sparkapp:1.0.0 phase: Failed status: [ContainerStatus(containerID=docker://3617a400e4604600d5fcc69df396facafbb2d9cd485a63bc324c1406e72f0d35, image=dockerhub.com/myapp/sparkapp:1.0.0, imageID=docker-pullable://dockerhub.com/sparkapp@sha256:f051d86384422dff3e8c8a97db823de8e62af3ea88678da4beea3f58cdb924e5, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://3617a400e4604600d5fcc69df396facafbb2d9cd485a63bc324c1406e72f0d35, exitCode=1, finishedAt=Time(time=2018-03-06T15:31:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-03-06T15:31:24Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})] 2018-03-06 10:31:28 INFO LoggingPodStatusWatcherImpl:54 - Container final statuses:

Container name: spark-kubernetes-driver Container image: myapp/sparkapp:1.0.0 Container state: Terminated Exit code: 1

Answer 1

下面是驱动程序 pod 提交的 spark 配置。我从 K8s UI 中提取了这个。 @TobiSH 让我知道这是否有助于解决我的问题。

SPARK_DRIVER_MEMORY: 1g
SPARK_DRIVER_CLASS: com.sparrkonk8.rdd.MockWordCount
SPARK_DRIVER_ARGS: 
SPARK_DRIVER_BIND_ADDRESS: 
SPARK_MOUNTED_CLASSPATH: /SparkApp.jar:/SparkApp.jar
SPARK_JAVA_OPT_0: -Dspark.kubernetes.executor.podNamePrefix=sparkapp-028d46fa109e309b8dfe1a4eceb46b61
SPARK_JAVA_OPT_1: -Dspark.app.name=sparkapp
SPARK_JAVA_OPT_2: -Dspark.kubernetes.driver.pod.name=sparkapp-028d46fa109e309b8dfe1a4eceb46b61-driver
SPARK_JAVA_OPT_3: -Dspark.executor.instances=5
SPARK_JAVA_OPT_4: -Dspark.submit.deployMode=cluster
SPARK_JAVA_OPT_5: -Dspark.driver.blockManager.port=7079
SPARK_JAVA_OPT_6: -Dspark.kubernetes.container.image=docker.com/myapp/sparkapp:1.0.0
SPARK_JAVA_OPT_7: -Dspark.app.id=spark-5e3beb5109174f40a84635b786789c30
SPARK_JAVA_OPT_8: -Dspark.master= k8s://https://k8-master
SPARK_JAVA_OPT_9: -Dspark.driver.host=sparkapp-028d46fa109e309b8dfe1a4eceb46b61-driver-svc.default.svc
SPARK_JAVA_OPT_10: -Dspark.jars=/opt/spark/work-dir/SparkApp.jar,/opt/spark/work-dir/SparkApp.jar
SPARK_JAVA_OPT_11: -Dspark.driver.port=7078`