java - Amazon EMR 中的 org.apache.hadoop.mapred.FileAlreadyExistsException

Question

我正在尝试在 Amazon EMR 中运行 WordCount 程序，但我收到错误消息:

Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory s3://mywordcountbuckett/run0 already exists
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at wordcount.WordCount.main(WordCount.java:70)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

我已经尝试在线谷歌解决方案并遇到了这样建议的解决方案

 The work around this is to not output to the top level of your bucket. So for instance "s3n://test.output/run0".

我试过了，还是不行。

Answer 1

“目录”是否存在于 s3 上？

如果是这样，您必须在再次运行 hadoop 作业之前将其删除。

您可以使用 s3cmd 或 aws cli 工具进行检查

s3cmd ls s3://mywordcountbuckett/run0/

aws s3 ls s3://mywordcountbuckett/run0/

删除数据(这会删除s3://mywordcountbuckett/run0/下的所有数据)

s3cmd del -r s3://mywordcountbuckett/run0/

aws s3 rm s3://mywordcountbuckett/run0/