java - 使用 Hadoop 进行 MapReduce， "Ouput file directory already exists"

Question

我正在尝试使用 Amazon EMR 服务上托管的集群。我正在尝试使用 WordCount.jar 文件和带有随机输出文件夹的 input1.txt 来运行 WordCount 示例。我的输入语法是 WordCount argv[1] argv[2]。 input1.txt 的路径为 argv[1]，argv[2] 是输出的路径。我知道如果您尝试多次使用相同的输出目录，hadoop 会生气。为了让你们绝对确定，我有台词 ->

FileInputFormat.addInputPath(job, new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));

但是无论我放入哪个输出目录，这都是我的结果，

Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory https://s3.us-east-2.amazonaws.com/mydirectory/gabagooba already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:141)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
at WordCountDemo.WordCount.main(WordCount.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)

我的输出目录应该存储在 S3 存储中，但是当我去那里时，没有输出。任何帮助都会让我摆脱困境。

Answer 1

能否指定输出位置为s3而不是 https。

示例:s3://yourbucket/output

引用号:https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-output.html