hadoop - hadoop distcp由于缺少 yarn 日志目录而失败

Question

我正在尝试在EMR群集上运行distcp命令:

hadoop distcp s3a://... hdfs://host/data/...

当我运行它时，它给出以下错误:

Exit code: 1
Exception message: /bin/bash: /mnt/yarn/logs/application_1524773139099_0003/container_1524773139099_0003_02_000001/stdout: No such file or directory

Stack trace: ExitCodeException exitCode=1: /bin/bash: /mnt/yarn/logs/application_1524773139099_0003/container_1524773139099_0003_02_000001/stdout: No such file or directory

at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

我已经检查了集群中的所有节点，它们都具有我创建的 /mnt/yarn/logs目录。这里发生了什么？

Answer 1

确保运行作业的用户具有足够的特权，可以在/ mnt / yarn / logs路径中创建临时目录，例如application _ *******。最好是配置单元用户，还可以提取application_1524773139099_0003的毛线记录，以查看可能解释实际错误的错误。