java - Spark集群与Spark客户端

Question

我是Spark的新手。我觉得我的问题与Spark的体系结构有关。一个应用程序(用Java开发)可以使用yarn-client模式在Yarn上运行。该命令是
./bin/spark-submit --verbose --class path.to.Class --master yarn-client --files lib.so --conf spark.yarn.jar=hdfs://master:8020/spark/spark-assembly-1.3.1-hadoop2.6.0.jar --num-executors 30 --driver-memory 512m --executor-memory 512m --executor-cores 3 myJarFile.jar
但是，如果我将模式更改为“yarn-cluster”，则会收到此错误。

Exception in thread "main" org.apache.spark.SparkException: Application application_1440071752950_0001 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:841) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:867) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

任何帮助将不胜感激!

非常感谢!

Answer 1

您提供的命令中缺少--jars选项。

引用spark文档:“--jars选项允许SparkContext.addJar函数与本地文件一起使用并以yarn-cluster模式运行。”