个人中心
gpt4 book ai didi

AWS Glue: How to write to S3 cross account(AWS Glue：如何写入S3交叉账户)

转载 作者：bug小助手 更新时间：2023-10-25 16:38:43 26 4
gpt4 key购买 nike



I have a Glue job in my account (Account-local), and I need to write the ETL output to another account (Account-remote). The job runs with an IAM role in local account: Glue-job-role

我的帐户(帐户本地)中有一个Glue作业，我需要将ETL输出写入另一个帐户(帐户远程)。该作业以本地帐户中的IAM角色运行：glue-job-ole


There is an IAM role in Account-remote, with name External-clients-role, which has policy providing permissions to S3 resource actions, such as: s3:PutObject, s3:GetObject etc to the S3 bucket in Account-remote.

Account-Remote中有一个IAM角色，名为External-Clients-Role，它具有向Account-Remote中的S3存储桶提供对S3资源操作的权限的策略，例如：S3：PutObject、S3：GetObject等。


Additionally, in the Trust Relationship of External-clients-role, the local glue job role Glue-job-role is added as a Principal, with sts:AssumeRole action, as:

此外，在外部客户-角色的信任关系中，添加本地胶水作业角色Glue-JOB-Role作为主体，带有sts：Assum eRole操作，如下：


//External-client-role trust relationship:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<account_number>:role/Glue-Job-Role"
                ]
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

i.e., its expected that the clients of Account-remote would assume the role, and use that to write to S3 bucket. However, I am not able to figure out how to assume this role.

即，预期Account-Remote的客户端将承担该角色，并使用该角色写入S3存储桶。然而，我不能想出如何承担这个角色。


Queries:

查询：


Since Glue-job is running with a local IAM role for ETL steps, can it assume External-clients-role just for the part of writing to remote S3 ? If yes, how to do it? Is there any other way ? enter image description here

由于Glue-JOB在ETL步骤中使用本地IAM角色运行，它是否可以仅针对写入远程S3的一部分承担外部客户端角色？如果是，怎么做？还有别的办法吗？


I tried looking at resources on how to assume the remote account role, however not able to find good lead on it.

我尝试查看有关如何承担远程帐户角色的资源，但找不到好的线索。


更多回答
优秀答案推荐

You should use Bucket policy for that case.

对于这种情况，您应该使用Bucket策略。



  1. Add following bucket policy to a bucket on remote account:


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGlueAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AWS-Local-account-id>:root"
      },
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<bucket-name>/*"
      ]
    }
  ]
}


  1. Grant access to this bucket for AWS Glue, normally using the IAM Role that is assigned to the task.

  2. Reuse your current Glue task code, as you don't need to change anything.


This assumes you can create a bucket policy, which might be not true, at the same time it's the simplest approach.

这假设您可以创建一个存储桶策略(这可能不是真的)，同时这是最简单的方法。



The second approach that would do it in a way, that you've tried, is to manually assume the role and overwrite credentials, however because I haven't tested it and I'm not sure if overwriting credentials is possible after reading data from data source, I'm only copy-pasting the link:


https://tmmr.uk/post/aws-glue-assume-temporary-credentials/

Https://tmmr.uk/post/aws-glue-assume-temporary-credentials/


更多回答

26 4 0
文章推荐： Neo4J: Renaming property keys(Neo4J：重命名属性键)
文章推荐： One big single redis instance for everything or multiple redis instances?(一个大的单一Redis实例用于所有内容，还是多个Redis实例？)
文章推荐： Create a single object from chained comparison in Python(在Python中通过链接比较创建单个对象)
文章推荐： Does the linesearch in scipy.optimize.minimize() with BFGS calculate gradients of the objective function for different values of the step size?(使用bfgs的scipy.Optimize.Minimum()中的行搜索是否为不同步长的值计算目标函数的梯度？)
bug小助手
个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章
滴滴打车优惠券免费领取
滴滴打车优惠券
全站热门文章
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com