java - 我在Windows中安装了Hadoop，但是当我运行hadoop示例失败时

Question

我准备在Windows上安装hadoop时，正在运行演示程序的4个守护程序正在运行

yarn jar %HADOOP_PREFIX%\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.5.0.jar wordcount /myfile.txt /out

这是我的env:jdk1.8，hadoop-2.7.7

我收到以下错误:

-_-

D:\ProgramFiles\hadoop\hadoop-2.7.7\share\hadoop\mapreduce>yarn jar hadoop-mapreduce-examples-2.7.7.jar wordcount /myfile.txt /input
19/03/02 10:05:11 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:19000/input already exists
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
        at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:141)

D:\ProgramFiles\hadoop\hadoop-2.7.7\share\hadoop\mapreduce>yarn jar hadoop-mapreduce-examples-2.7.7.jar wordcount /myfile.txt /input1
19/03/02 10:06:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/03/02 10:06:26 INFO input.FileInputFormat: Total input paths to process : 1
19/03/02 10:06:27 INFO mapreduce.JobSubmitter: number of splits:1
19/03/02 10:06:27 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1551490342050_0002
19/03/02 10:06:27 INFO impl.YarnClientImpl: Submitted application application_1551490342050_0002
19/03/02 10:06:27 INFO mapreduce.Job: The url to track the job: http://SF80002935M01:8088/proxy/application_1551490342050_0002/
19/03/02 10:06:27 INFO mapreduce.Job: Running job: job_1551490342050_0002
19/03/02 10:07:02 INFO mapreduce.Job: Job job_1551490342050_0002 running in uber mode : false
19/03/02 10:07:02 INFO mapreduce.Job:  map 0% reduce 0%
19/03/02 10:07:02 INFO mapreduce.Job: Job job_1551490342050_0002 failed with state FAILED due to: Application application_1551490342050_0002 failed 2 times due to AM Container for appattempt_1551490342050_0002_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://SF80002935M01:8088/cluster/app/application_1551490342050_0002Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1551490342050_0002_02_000001
Exit code: 1
Exception message: CreateSymbolicLink error (5): ?????



Stack trace: ExitCodeException exitCode=1: CreateSymbolicLink error (5): ?????



        at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
        at org.apache.hadoop.util.Shell.run(Shell.java:482)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)

Shell output: 移动了         1 个文件。


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
19/03/02 10:07:02 INFO mapreduce.Job: Counters: 0

D:\ProgramFiles\hadoop\hadoop-2.7.7\share\hadoop\mapreduce>hdfs dfs -cat /input1
cat: `/input1': No such file or directory

D:\ProgramFiles\hadoop\hadoop-2.7.7\share\hadoop\mapreduce>hdfs dfs -cat /input1 -r -00000
cat: `/input1': No such file or directory
cat: `-r': No such file or directory
cat: `-00000': No such file or directory

Answer 1

hdfs://localhost:19000/input already exists

看来您给 /input作为 /myfile.txt字数的 输出目录

如果MapReduce作业已经存在，它将不会覆盖输出目录的内容，因此将失败。

至于底部的错误， /input1将是一个目录，而您不能 cat是一个目录，则 -r和 -00000分别是无效的输出文件名。

而是执行 hdfs dfs -ls /input1，然后将文件名之一复制到 cat