hadoop - 在 S3 中使用 hadoop 命令时需要哪些策略

Question

我想将on-pre中的数据复制到S3。

我尝试为此使用以下命令。

hadoop fs -Dfs.s3a.access.key=******* -Dfs.s3a.secret.key=******* -cp -f hdfs://on-pre/cluster/mydata/dt=20200601/ s3a://some-bucket/somewhere/

当我运行这个命令时，我得到以下错误(路径都是假的):

cp: s3a://some-bucket/somewhere/dt=20200601/000000_0.gz: getFileStatus on s3a://some-bucket/somewhere/dt=20200601/000000_0.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxx

我设置了以下 S3 策略。

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::some-bucket/somewhere/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": "s3:List*",
            "Resource": "arn:aws:s3:::some-bucket",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "somewhere/*"
                    ]
                }
            }
        }
    ]
}

我应该设置什么 S3 策略来使用 hadoop fs cp ?

Answer 1

子目录本身需要 ListBucket

如评论中所述，必要的权限可能会更改。

但是，这是解决它的代码供您引用

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::some-bucket/somewhere/*"
            ]
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::some-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "somewhere",
                        "somewhere/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::some-bucket"
            ]
        }
    ]
}

注意 此 IAM 声明于 2020 年 6 月 5 日适用于 Hadoop 3.2.1 或更早版本。 future 的 Hadoop 版本可能会更改规则，因为它们或 AWS 会分别更改连接器或 S3 的功能。