apache - Tez 在 HDP 2.1 上崩溃

Question

当我尝试在终端中运行 TEZ 作业时，我的配置单元崩溃了。我在一个有 5 个节点的集群上安装了 HDP 2.1。操作系统:红帽企业

谁能帮帮我？ :)

来自应用程序跟踪器的日志:

HDP: 2.1
User: hdfs
Name: HIVE-9fd632e5-d76a-4e0a-8845-0c1f0af52f5c
Application Type: TEZ
Application Tags:
State: FAILED
FinalStatus: FAILED
Started: 1-Jul-2014 12:03:22
Elapsed: 4sec
Tracking URL: History
Diagnostics:
Application application_1404208879507_0002 failed 2 times due to AM Container for    appattempt_1404208879507_0002_000002 exited with exitCode: 1 due to: Exception from container-    launch:     org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at         org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecu    tor.java:195)
at     org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch    .java:300)
at     org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch    .java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.

在终端中:

Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

Answer 1

可能是运行job的用户id不能小于1000，去TaskTracker的日志文件看是否有类似这样的信息:

INFO org.apache.hadoop.mapred.TaskController: Reading task con­troller configuration /etc/hadoop/taskcontroller.cfg
INFO. org.apache.hadoop.mapred.TaskController: requested user
hdfs has id 201, which is below the minimum allowed 1000 

如果是这种情况，那么您可以执行以下操作:

usermod -u 10000 hdfs