apache-spark - Spark的示例在客户端模式下引发FileNotFoundException

Question

我有:Ubuntu 14.04，Hadoop 2.7.7，Spark 2.2.0。

我刚刚安装了所有东西。

当我尝试运行Spark的示例时:

bin/spark-submit --deploy-mode client \
               --class org.apache.spark.examples.SparkPi \
               examples/jars/spark-examples_2.11-2.2.0.jar 10

我收到以下错误:

INFO yarn.Client: client token: N/A diagnostics: Application application_1552490646290_0007 failed 2 times due to AM Container for appattempt_1552490646290_0007_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://ip-123-45-67-89:8088/cluster/app/application_1552490646290_0007 Then, click on links to logs of each attempt. Diagnostics: File file:/tmp/spark-f5879f52-6777-481a-8ecf-bbb55e376901/__spark_libs__6948713644593068670.zip does not exist java.io.FileNotFoundException: File file:/tmp/spark-f5879f52-6777-481a-8ecf-bbb55e376901/__spark_libs__6948713644593068670.zip does not exist

 at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:421)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)

在客户端模式和群集模式下，我都遇到相同的错误。

Answer 1

似乎通过加载spark库失败。正如Daniel指出的那样，这可能与您的阅读权利有关。此外，这可能与磁盘空间不足有关。

但是，在本例中，为了避免向本地计算机的主服务器传输延迟和读取/写入权限，我们将 spark-libs 放入Yarn群集的HDFS中，然后将它们指向spark.yarn.archive属性。

jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
export HADOOP_USER_NAME=hadoop
hadoop fs -mkdir -p /apps/spark/
hadoop fs -put -f  ${SPARK_HOME}/spark-libs.jar /apps/spark/

# spark-defaults.conf
spark.yarn.archive hdfs:///apps/spark/spark-libs.jar