amazon-web-services - AWS-EMR : Where is log file?

Question

我正在使用 EMR 通过 yarn 运行 spark 作业。作业失败但我找不到 EMR 记录异常的位置，我可以在主节点屏幕输出上看到回溯，如下所示。应该有一个更详细的日志文件来显示导致异常的原因，但我找不到它在哪里。我查看了没有显示任何错误的 hdfs://var/log/spark/app/application_xxx。这是我提交申请的方式:

spark-submit --deploy-mode cluster --master yarn --num-executors 1 --executor-cores 2 --executor-memory 5g word2vec_app.py hdfs:///test/r8_no_sto.txt

这是主节点上的屏幕输出:

Exception in thread "main" org.apache.spark.SparkException: Application application_1488419676573_0005 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

节点的stdout/stderr可以在每个节点的/mnt/var/log/hadoop-yarn/containers/application中找到。