apache-spark - Spark on K8s - 出现错误 : kube mode not support referencing app depenpendcies in local

Question

我正在尝试在 k8s 上设置一个 Spark 集群。我已经按照这篇文章创建并设置了一个包含三个节点的集群:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

之后，当我尝试在集群上部署 spark 时，它在 spark 提交设置中失败了。
我使用了这个命令:

~/opt/spark/spark-2.3.0-bin-hadoop2.7/bin/spark-submit \
--master k8s://https://206.189.126.172:6443 \
--deploy-mode cluster \
--name word-count \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=docker.io/garfiny/spark:v2.3.0 \
—-conf spark.kubernetes.driver.pod.name=word-count \
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar

它给了我这个错误:

Exception in thread "main" org.apache.spark.SparkException: The Kubernetes mode does not yet support referencing application dependencies in the local file system.
    at org.apache.spark.deploy.k8s.submit.DriverConfigOrchestrator.getAllConfigurationSteps(DriverConfigOrchestrator.scala:122)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:229)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:227)
    at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2585)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:227)
    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:192)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

2018-06-04 10:58:24 INFO ShutdownHookManager:54 - 关闭 Hook 被调用
2018-06-04 10:58:24 INFO ShutdownHookManager:54 - 删除目录/private/var/folders/lz/0bb8xlyd247cwc3kvh6pmrz00000gn/T/spark-3967f4ae-e8b3-20c-429dc

注意:我按照这篇文章在 k8s 上安装 spark。
https://spark.apache.org/docs/latest/running-on-kubernetes.html

Answer 1

错误信息来自 commit 5d7c4ba4d73a72f26d591108db3c20b4a6c84f3f并包括您提到的页面:“Running Spark on Kubernetes ”以及您指出的提及:

// TODO(SPARK-23153): remove once submission client local dependencies are supported.
if (existSubmissionLocalFiles(sparkJars) || existSubmissionLocalFiles(sparkFiles)) {
  throw new SparkException("The Kubernetes mode does not yet support referencing application " +
    "dependencies in the local file system.")
}

这在 SPARK-18278 中有描述:

it wouldn't accept running a local: jar file, e.g. local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar, on my spark docker image (allowsMixedArguments and isAppResourceReq booleans in SparkSubmitCommandBuilder.java get in the way).

这与 kubernetes issue 34377 相关联

issue SPARK-22962 "Kubernetes app fails if local files are used"提到:

This is the resource staging server use-case. We'll upstream this in the 2.4.0 timeframe.

同时，该错误消息是在 PR 20320 中引入的。 .

它包括评论:

The manual tests I did actually use a main app jar located on gcs and http.
To be specific and for record, I did the following tests:

• Using a gs:// main application jar and a http:// dependency jar. Succeeded.
• Using a https:// main application jar and a http:// dependency jar. Succeeded.
• Using a local:// main application jar. Succeeded.
• Using a file:// main application jar. Failed.
• Using a file:// dependency jar. Failed.

这个问题现在应该已经解决了， OP garfiny确认 in the comments :

I used the newest spark-kubernetes jar to replace the one in spark-2.3.0-bin-hadoop2.7 package. The exception is gone.