hadoop - 当查看在 yarn 上运行的已完成 spark 作业的日志时重定向到容器的日志服务器

Question

我在 yarn 上运行 spark。

我的spark版本是2.1.1，hadoop版本是apache hadoop 2.7.3。

当 spark 作业以集群模式在 yarn 上运行时，我可以通过 stdout/stderr 链接查看 Executor 的日志，如

http://hadoop-slave1:8042/node/containerlogs/container_1500432603585_0148_01_000001/hadoop/stderr?start=-4096

但是当作业完成时，通过 stdout/stderr 链接查看 Executor 的日志将得到类似

的错误页面

Redirecting to log server for container_1500432603585_0148_01_000001

java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all.

然后它会自动重定向到

http://hadoop-slave1:8042/node/hadoop-master:19888/jobhistory/logs/hadoop-slave1:36207/container_1500432603585_0148_01_000001/container_1500432603585_0148_01_000001/hadoop

并获取其他错误页面，例如

Sorry, got error 404  
Please consult RFC 2616 for meanings of the error code.  

Error Details

org.apache.hadoop.yarn.webapp.WebAppException: /hadoop-master:19888/jobhistory/logs/hadoop-slave1:50284/container_1500432603585_0145_01_000002/container_1500432603585_0145_01_000002/oryx: controller for hadoop-master:19888 not found
        at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:232)
        at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:140)
        at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:134)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)

实际上，当 Spark 作业完成:

http://hadoop-master:19888/jobhistory/logs/hadoop-slave1:36207/container_1500432603585_0148_01_000001/container_1500432603585_0148_01_000001/hadoop

它与之前的url有点不同，它去掉了头部“hadoop-slave1:8042/node/”。

有谁知道在 spark 作业完成时查看 spark 日志的另一种更好的方法？

我已经配置了 yarn-site.xml

  <property>
      <name>yarn.resourcemanager.hostname</name>
      <value>hadoop-master</value>
      <description>The hostname of the RM.</description>
  </property>  

  <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
  </property>

  <property>
      <name>yarn.log.server.url</name>
      <value>${yarn.resourcemanager.hostname}:19888/jobhistory/logs</value>
  </property>

和 mapred-site.xml

<property>
    <name>mapreduce.jobhistory.address</name>
    <value>${yarn.resourcemanager.hostname}:10020</value>
  </property>

  <property>
    <name>mapreduce.jobhistory.admin.address </name>
    <value>${yarn.resourcemanager.hostname}:10033</value>
  </property>

  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>${yarn.resourcemanager.hostname}:19888</value>
  </property>

Answer 1

我遇到过这种情况。通过 YARN UI History 选项卡查看已完成的 spark steaming 作业日志，但出现以下错误:

Failed while trying to construct the redirect url to the log server. Log Server url may not be configured java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all.

解决方案是配置文件 yarn-site.xml。添加 key yarn.log.server.url :

<property>
<name>yarn.log.server.url</name>        
<value>http://<LOG_SERVER_HOSTNAME>:19888/jobhistory/logs</value>
</property>

然后重启yarn cluster重新加载yarn-site.xml。(这一步很重要!)