java - Spark 2.6.0 : Exception in thread "main" java. lang.ClassNotFoundException: 当 位于提交到集群的 jar 中时

Question

我可以在本地运行我用 Scala 编写的 Spark 应用程序:

sbt run ...

我让它从命令行运行良好，没有错误。

当我将我在 Scala 中编写的 Spark 应用“spark-submit”到 2.6.0 集群时，如下所示:

spark-submit --class MyTest --master spark://my-spark-01a:7077 --deploy-mode cluster --supervise --executor-memory 20G --total-executor-cores 100 --jars $IGNITE_JARS,/home/ubuntu/tmp/mytest-assembly-0.3.1.1.jar  /home/ubuntu/tmp/mytest-assembly-0.3.1.1.jar

...我在 Spark 集群的 stderr 中看到以下错误

18/12/12 00:41:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/12/12 00:41:24 INFO spark.SecurityManager: Changing view acls to: ubuntu
18/12/12 00:41:24 INFO spark.SecurityManager: Changing modify acls to: ubuntu
18/12/12 00:41:24 INFO spark.SecurityManager: Changing view acls groups to: 
18/12/12 00:41:24 INFO spark.SecurityManager: Changing modify acls groups to: 
18/12/12 00:41:24 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ubuntu); groups with view permissions: Set(); users  with modify permissions: Set(ubuntu); groups with modify permissions: Set()
18/12/12 00:41:24 INFO util.Utils: Successfully started service 'Driver' on port 50983.
18/12/12 00:41:24 INFO worker.WorkerWatcher: Connecting to worker spark://Worker@127.0.0.1:44303
Exception in thread "main" java.lang.ClassNotFoundException: MyTest

“MyTest”类位于我的/home/ubuntu/tmp/mytest- assembly-0.3.1.1.jar

我就是不明白——这没有意义。为什么 Spark 提示我的主类在我正在部署的 jar 中找不到？我还忘记了什么(其他)让这件事正常工作？

请帮忙。

Answer 1

当您在集群模式下提交时，您可能需要将此 jar 传递给驱动程序和执行程序。添加后再次尝试spark-submit；

--conf spark.driver.extraClassPath=/your/file.jar
--conf spark.executor.extraClassPath=/your/file.jar