amazon-web-services - AWS EMR 集群流式处理步骤 : Bad Request

Question

我正在尝试设置一个简单的 EMR 作业来对存储在 s3://__mybucket__/input/ 中的大量文本文件执行字数统计。我无法正确添加两个必需的流式处理步骤中的第一个(第一个是将输入映射到 wordSplitter.py，使用 IdentityReducer 减少到临时存储；第二个步骤是使用 /bin/wc/ 映射此辅助存储的内容，并再次使用 IdentityReducer 进行缩减。

这是第一步的(失败)描述:

Status:FAILED
Reason:S3 Service Error.
Log File:s3://aws-logs-209733341386-us-east-1/elasticmapreduce/j-2XC5AT2ZP48FJ/steps/s-1SML7U7CXRDT5/stderr.gz
Details:Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 7799087FCAE73457), S3 Extended Request ID: nQYTtW93TXvi1G8U4LLj73V1xyruzre+uSt4KN1zwuIQpwDwa+J8IujOeQMpV5vRHmbuKZLasgs=
JAR location: command-runner.jar
Main class: None
Arguments: hadoop-streaming -files s3://elasticmapreduce/samples/wordcount/wordSplitter.py -mapper wordSplitter.py -reducer org.apache.hadoop.mapred.lib.IdentityReducer -input s3://__mybucket__/input/ -output s3://__mybucket__/output/
Action on failure: Continue

这是发送到 hadoop 集群的命令:

JAR location : command-runner.jar
Main class : None
Arguments : hadoop-streaming -mapper s3a://elasticmapreduce/samples/wordcount/wordSplitter.py -reducer aggregate -input s3a://__my_bucket__/input/ -output s3a://__my_bucket__/output/

Answer 1

我认为这里的解决方案可能非常简单。

不要使用 s3://，而是使用 s3a:// 作为您的作业访问存储桶的方案。参见 here ，s3:// 方案已弃用，并且要求相关存储桶专用于您的 Hadoop 数据。引用上述文档链接:

This filesystem requires you to dedicate a bucket for the filesystem - you should not use an existing bucket containing files, or write other files to the same bucket. The files stored by this filesystem can be larger than 5GB, but they are not interoperable with other S3 tools.