apache-spark - 如何在Kubernetes上将作业发送到Spark外部调度程序无法实例化

Question

我已经启动了一个Minikube集群，但是当我提交spark-submit时，作业会出错。

我的命令:

bin/spark-submit --master k8s://https://192.168.99.101:8443 --deploy-mode cluster --name spark-test --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar

我收到以下错误:

2019-03-13 18:26:57 ERROR SparkContext:91 - Error initializing SparkContext. org.apache.spark.SparkException: External scheduler cannot be instantiated at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) ...30 more Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/pods/spark-test-1552501611130-driver. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-test-1552501611130-driver" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default". ... 30 more

Answer 1

您需要在Kubernetes中创建ServiceAccount。如果您从此link下载YAML配置，则可以完成此操作

然后跑

kubectl apply -f spark-rbac.yaml

YAML文件将创建命名空间和ServiceAccount，这是Kubernetes上Spark所需的。
但是，您还需要修改spark-submit命令，使其类似于:

bin/spark-submit --master k8s://https://192.168.99.101:8443 --deploy-mode cluster --name spark-test --class org.apache.spark.examples.SparkPi \
--conf spark.kubernetes.namespace=spark \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar