python-3.x - 使用 boto3 连接到 S3 加速端点

Question

我想从已激活加速的 S3 存储桶中将文件下载到 Python 文件对象中。我遇到了一些建议是否覆盖 endpoint_url 的资源。到“s3-accelerate.amazonaws.com”和/或使用 use_accelerate_endpoint属性。

我已经尝试了这两种方法，并尝试了几种变体，但每次都返回相同的错误。我试过的脚本之一是:

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "path"}))
   input = BytesIO() 
   s3.download_fileobj(<MY_BUCKET>,<MY_KEY>, input)

返回以下错误:

            ---------------------------------------------------------------------------
            ClientError                               Traceback (most recent call last)
            <ipython-input-61-92b89b45f215> in <module>()
                 11      "addressing_style": "path"}))
                 12 input = BytesIO()
            ---> 13 s3.download_fileobj(bucket, filename, input)
                 14 
                 15 

            ~/Project/venv/lib/python3.5/site-packages/boto3/s3/inject.py in download_fileobj(self, Bucket, Key, Fileobj, ExtraArgs, Callback, Config)
                568             bucket=Bucket, key=Key, fileobj=Fileobj,
                569             extra_args=ExtraArgs, subscribers=subscribers)
            --> 570         return future.result()
                571 
                572 

            ~/Project//venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                 71             # however if a KeyboardInterrupt is raised we want want to exit
                 72             # out of this and propogate the exception.
            ---> 73             return self._coordinator.result()
                 74         except KeyboardInterrupt as e:
                 75             self.cancel()

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/futures.py in result(self)
                231         # final result.
                232         if self._exception:
            --> 233             raise self._exception
                234         return self._result
                235 

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/tasks.py in _main(self, transfer_future, **kwargs)
                253             # Call the submit method to start submitting tasks to execute the
                254             # transfer.
            --> 255             self._submit(transfer_future=transfer_future, **kwargs)
                256         except BaseException as e:
                257             # If there was an exception raised during the submission of task

            ~/Project/venv/lib/python3.5/site-packages/s3transfer/download.py in _submit(self, client, config, osutil, request_executor, io_executor, transfer_future)
                347                 Bucket=transfer_future.meta.call_args.bucket,
                348                 Key=transfer_future.meta.call_args.key,
            --> 349                 **transfer_future.meta.call_args.extra_args
                350             )
                351             transfer_future.meta.provide_transfer_size(

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
                310                     "%s() only accepts keyword arguments." % py_operation_name)
                311             # The "self" in this scope is referring to the BaseClient.
            --> 312             return self._make_api_call(operation_name, kwargs)
                313 
                314         _api_call.__name__ = str(py_operation_name)

            ~/Project/venv/lib/python3.5/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
                603             error_code = parsed_response.get("Error", {}).get("Code")
                604             error_class = self.exceptions.from_code(error_code)
            --> 605             raise error_class(parsed_response, operation_name)
                606         else:
                607             return parsed_response

            ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

当我使用 "use_accelerate_endpoint": False 运行相同的脚本时它工作正常。
但是，它在以下情况下返回相同的错误:

我覆盖了 endpoint_url与“s3-accelerate.amazonaws.com”

我定义 "addressing_style": "virtual"

运行时 s3.get_bucket_accelerate_configuration(Bucket=<MY_BUCKET>)
我收到 {..., 'Status': 'Enabled'}正如预期的那样。

知道该代码有什么问题以及我应该更改什么以正确查询该存储桶的加速端点吗？

在 Ubuntu 17.04 上使用带有 boto3==1.4.7、botocore==1.7.43 的 python3.5。

编辑:
我也尝试过类似的上传脚本:

    from botocore.config import Config
    import boto3
    from io import BytesIO
    session = boto3.session.Session()
    s3 = session.client(                                                                            
           service_name='s3',                                                                  
           aws_access_key_id=<MY_KEY_ID>,               
           aws_secret_access_key=<MY_KEY>,
           region_name="us-west-2",
       config=Config(s3={"use_accelerate_endpoint": True,
                         "addressing_style": "virtual"}))
   output = BytesIO()
   output.seek(0)
   s3.upload_fileobj(output, <MY_BUCKET>,<MY_KEY>)

在没有 use_accelerate_endpoint 的情况下工作选项(所以我的键没问题)，但在 True 时返回此错误:

   ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

我都试过了 addressing_style此处也有选项(虚拟和路径)

Answer 1

使用 boto3==1.4.7和 botocore==1.7.43 .
这是从启用了传输加速的存储桶中检索对象的一种方法。

import boto3
from botocore.config import Config
from io import BytesIO

config = Config(s3={"use_accelerate_endpoint": True})
s3_resource = boto3.resource("s3",
                             aws_access_key_id=<MY_KEY_ID>,               
                             aws_secret_access_key=<MY_KEY>,
                             region_name="us-west-2",
                             config=config)

s3_client = s3_resource.meta.client

file_object = BytesIO()
s3_client.download_fileobj(<MY_BUCKET>, <MY_KEY>, file_object)

请注意 客户端发送 HEAD在 GET 之前向加速端点发出请求.

其规范请求类似于以下内容:

CanonicalRequest:
HEAD
/<MY_KEY>

host:<MY_BUCKET>.s3-accelerate.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200520T204128Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

HEAD的一些原因请求可能失败包括:

具有给定键的对象不存在或启用了严格的访问控制

凭据无效

传输加速未启用