hadoop - 如何用 yarn 进行elasticsearch？

Question

我的hadoop版本是2.5.0

遵循指南http://www.elastic.co/guide/en/elasticsearch/hadoop/2.1.Beta/ey-usage.html

我下载了elasticsearh-hadoop-2.1.0beta4.zip。

我以波纹管的方式运行hadoop jar elasticsearh-yarn-2.1.0beta4.jar。

hadoop jar elasticsearh-yarn-2.1.0beta4.jar -download-es 
hadoop jar elasticsearh-yarn-2.1.0beta4.jar -install-es
hadoop jar elasticsearh-yarn-2.1.0beta4.jar -install

elastic-1.5.2已下载。
elastic-1.5.2.zip和elasticsearh-yarn-2.1.0beta4.jar已上传到hdfs:// host:port / apps / elasticsearch。

当我用yarn启动es时，输出很好。

hadoop jar /tmp/elasticsearch-hadoop-2.1.0.Beta4/dist/elasticsearch-yarn-2.1.0.Beta4.jar -start
15/05/13 16:50:09 INFO client.RMProxy: Connecting to ResourceManager at hdp1/10.80.3.231:8032
15/05/13 16:50:11 INFO impl.YarnClientImpl: Submitted application application_1431506015778_0003
Launched a 1 node Elasticsearch-YARN cluster [application_1431506015778_0003@http://xxxx:8088/proxy/application_1431506015778_0003/] at ......

但是当我稍后检查状态时，该应用程序失败。

hadoop jar /tmp/logsystem/elasticsearch-hadoop-2.1.0.Beta4/dist/elasticsearch-yarn-2.1.0.Beta4.jar -status
INFO client.RMProxy: Connecting to ResourceManager at hdp1/10.80.3.231:8032
Id                              State       Status     Start Time         Finish Time        Tracking URL
application_1431506015778_0001  FINISHED    FAILED     15-5-13 PM 4:36     15-5-13 下午4:37     http://xxx:8088/proxy/application_1431506015778_0001/A
application_1431506015778_0002  FINISHED    FAILED     15-5-13 PM 4:49     15-5-13 下午4:49     http://xxxx:8088/proxy/application_1431506015778_0002/A

我用纱检查了日志，AM的日志被吹了

15/05/13 16:50:18 INFO am.ApplicationMaster: Starting ApplicationMaster...
15/05/13 16:50:19 INFO client.RMProxy: Connecting to ResourceManager at hdp1/10.80.3.231:8030
15/05/13 16:50:19 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
15/05/13 16:50:19 INFO am.EsCluster: Allocating Elasticsearch cluster with 1 nodes
15/05/13 16:50:24 INFO impl.AMRMClientImpl: Received new token for : hdp4:8041
15/05/13 16:50:26 INFO am.EsCluster: About to launch container for command: [{{SHELL}} elasticsearch-1.5.2.zip/elasticsearch-1.5.2/bin/elasticsearch 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr]
15/05/13 16:50:26 INFO impl.ContainerManagementProtocolProxy: Opening proxy : hdp4:8041
15/05/13 16:50:26 INFO am.EsCluster: Started container Container: [ContainerId: container_1431506015778_0003_01_000002, NodeId: xxxx:8041, NodeHttpAddress: hdp4:8042, Resource: <memory:2048, vCores:1>, Priority: -1, Token: Token { kind: ContainerToken, service: XXXX:8041 }, ]
15/05/13 16:50:26 INFO am.EsCluster: Fully allocated 1 containers
**15/05/13 16:50:31 WARN am.EsCluster: Container container_1431506015778_0003_01_000002 exited with an invalid/unknown exit code...**
15/05/13 16:50:31 WARN am.EsCluster: Cluster has not completed succesfully...
15/05/13 16:50:31 INFO am.EsCluster: Cluster has completed running...
15/05/13 16:50:46 INFO impl.ContainerManagementProtocolProxy: Opening proxy : hdp4:8041
15/05/13 16:50:46 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.

在nodemanager的日志中，唯一的错误信息是exitcode 3

2015-05-13 16:50:26,083 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1431506015778_0003_01_000002 is : 3
2015-05-13 16:50:26,084 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1431506015778_0003_01_000002 and exit code: 3
ExitCodeException exitCode=3:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-05-13 16:50:26,085 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.

我的问题是:

1，es-hadoop 2.1.0beta4和es 1.5.2是否可以与hadoop-2.5.0一起运行？

2，我需要用hadoop-2.5.0重建jar吗？

3，exitcode 3是什么意思？

4，如何获取有关Container中错误的更多信息？

Answer 1

尝试查看应用程序日志，这些日志通常包含更多信息:

yarn logs -applicationId application_1431506015778_0002