scala - SparkPi 程序在 Yarn/Spark/Google Compute Engine 下保持运行

Question

在 Google Compute Engine 上部署了一个 Hadoop (Yarn + Spark) 集群，其中有一个主节点和两个从节点。当我运行以下 shell 脚本时:

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 1 --driver-memory 1g --executor-memory 1g --executor-cores 1/home/hadoop/spark-install/lib/spark-examples-1.1.0-hadoop2.4.0.jar 10

作业一直在运行，每一秒我都会收到类似这样的消息:

---

15/02/06 22:47:12 INFO yarn.Client: Application report from ResourceManager:
         application identifier: application_1423247324488_0008<br>
         appId: 8<br>
         clientToAMToken: null<br>
         appDiagnostics:<br>
         appMasterHost: hadoop-w-zrem.c.myapp.internal<br>
         appQueue: default<br>
         appMasterRpcPort: 0<br>
         appStartTime: 1423261517468<br>
         yarnAppState: RUNNING<br>
         distributedFinalState: UNDEFINED<br>
         appTrackingUrl: http://hadoop-m-xxxx:8088/proxy/application_1423247324488_0008/<br>
         appUser: achitre

Answer 1

使用--master yarn-client代替--master yarn-cluster