django - 使用django-storages和boto3在Django中调用CreateMultipartUpload操作时访问AccessDenied

Question

我想使用django-storages将我的模型文件存储在Amazon S3中，但是出现Access Denied错误。我已经向用户授予了对所有资源的几乎所有S3权限PutObject，ListBucketMultipartUploads，ListMultipartUploadParts，AbortMultipartUpload权限等，但这并不能解决。

settings.py

...
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_S3_REGION_NAME = 'eu-west-1'
AWS_S3_CUSTOM_DOMAIN = 'www.xyz.com'
AWS_DEFAULT_ACL = None
AWS_STORAGE_BUCKET_NAME = 'www.xyz.com'
...

使用Django Shell，我尝试使用存储系统，如下所示。

Python 3.6.6 (default, Sep 12 2018, 18:26:19)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import os
>>> AWS_ACCESS_KEY_ID = os.environ.get( 'AWS_ACCESS_KEY_ID', 'anything' )
>>> AWS_SECRET_ACCESS_KEY = os.environ.get( 'AWS_SECRET_ACCESS_KEY', 'anything' )
>>> AWS_DEFAULT_ACL = 'public-read'
>>> from django.core.files.storage import default_storage
>>> file = default_storage.open('test', 'w')
...
>>> file.write('storage contents')
2018-09-27 16:41:42,596 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function validate_ascii_metadata at 0x7fdb5e848d08>
2018-09-27 16:41:42,596 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function sse_md5 at 0x7fdb5e848158>
2018-09-27 16:41:42,597 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function validate_bucket_name at 0x7fdb5e8480d0>
2018-09-27 16:41:42,597 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateMultipartUpload: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fdb5c5d1128>>
2018-09-27 16:41:42,597 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function generate_idempotent_uuid at 0x7fdb5e846c80>
2018-09-27 16:41:42,598 botocore.hooks [DEBUG] Event before-call.s3.CreateMultipartUpload: calling handler <function add_expect_header at 0x7fdb5e848598>
2018-09-27 16:41:42,598 botocore.hooks [DEBUG] Event before-call.s3.CreateMultipartUpload: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fdb5c5d1128>>
2018-09-27 16:41:42,598 botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateMultipartUpload) with params: {'url_path': '/www.xyz.com/test?uploads', 'query_string': {}, 'method': 'POST', 'headers': {'Content-Type': 'application/octet-stream', 'User-Agent': 'Boto3/1.7.80 Python/3.6.6 Linux/4.14.67-66.56.amzn1.x86_64 Botocore/1.11.1 Resource'}, 'body': b'', 'url': 'https://s3.eu-west-1.amazonaws.com/www.xyz.com/test?uploads', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7fdb5c8e80b8>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'www.xyz.com'}}}
2018-09-27 16:41:42,599 botocore.hooks [DEBUG] Event request-created.s3.CreateMultipartUpload: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fdb5c8db780>>
2018-09-27 16:41:42,599 botocore.hooks [DEBUG] Event choose-signer.s3.CreateMultipartUpload: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fdb5cabff98>>
2018-09-27 16:41:42,599 botocore.hooks [DEBUG] Event choose-signer.s3.CreateMultipartUpload: calling handler <function set_operation_specific_signer at 0x7fdb5e846b70>
2018-09-27 16:41:42,599 botocore.hooks [DEBUG] Event before-sign.s3.CreateMultipartUpload: calling handler <function fix_s3_host at 0x7fdb5e983048>
2018-09-27 16:41:42,600 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.eu-west-1.amazonaws.com/www.xyz.com/test?uploads
2018-09-27 16:41:42,600 botocore.utils [DEBUG] Not changing URI, bucket is not DNS compatible: www.xyz.com
2018-09-27 16:41:42,601 botocore.auth [DEBUG] Calculating signature using v4 auth.
2018-09-27 16:41:42,601 botocore.auth [DEBUG] CanonicalRequest:
POST
/www.xyz.com/test
uploads=
content-type:application/octet-stream
host:s3.eu-west-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf343ddd27ae41e4649b934ca495991b7852b855
x-amz-date:20180927T164142Z

content-type;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afb65gdfg33441e4649b934ca495991b7852b855
2018-09-27 16:41:42,601 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20180927T164142Z
20180927/eu-west-1/s3/aws4_request
8649ef591fb64412e923359a4sfvvffdd6d00915b9756d1611b38e346ae
2018-09-27 16:41:42,602 botocore.auth [DEBUG] Signature:
61db9afe5f87730a75692af5a95ggffdssd6f4e8e712d85c414edb14f
2018-09-27 16:41:42,602 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://s3.eu-west-1.amazonaws.com/www.xyz.com/test?uploads, headers={'Content-Type': b'application/octet-stream', 'User-Agent': b'Boto3/1.7.80 Python/3.6.6 Linux/4.14.67-66.56.amzn1.x86_64 Botocore/1.11.1 Resource', 'X-Amz-Date': b'20180927T164142Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fbdsdsffdss649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=X1234567890/20180927/eu-west-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=61db9afe5f87730a7sdfsdfs20b7137cf5d6f4e8e712d85c414edb14f', 'Content-Length': '0'}>
2018-09-27 16:41:42,638 botocore.parsers [DEBUG] Response headers: {'x-amz-request-id': '9E879E78E4883471', 'x-amz-id-2': 'ZkCfOMwLoD08Yy4Nzfxsdfdsdfds3y9wLxzqFw+o3175I+QEdtdtAi8vIEH1vi9iq9VGUC98GqlE=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 27 Sep 2018 16:41:42 GMT', 'Server': 'AmazonS3'}
2018-09-27 16:41:42,639 botocore.parsers [DEBUG] Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>9E879E78E4883471</RequestId><HostId>ZkCfOMwLoD08Yy4Nzfxo8RpzsdfsdfsxzqFw+o3175I+QEdtdtAi8vIEH1vi9iq9VGUC98GqlE=</HostId></Error>'
2018-09-27 16:41:42,639 botocore.hooks [DEBUG] Event needs-retry.s3.CreateMultipartUpload: calling handler <botocore.retryhandler.RetryHandler object at 0x7fdb5c618ac8>
2018-09-27 16:41:42,640 botocore.retryhandler [DEBUG] No retry needed.
2018-09-27 16:41:42,640 botocore.hooks [DEBUG] Event needs-retry.s3.CreateMultipartUpload: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fdb5c5d1128>>
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/storages/backends/s3boto3.py", line 127, in write
    self._multipart = self.obj.initiate_multipart_upload(**parameters)
  File "/usr/local/lib/python3.6/dist-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied

这些是我正在使用的版本。

boto3==1.7.80
botocore==1.11.1
Django==2.1
s3transfer==0.1.13
django-storages==1.7.1

为什么会引发异常？

Answer 1

事实证明，我必须指定一个策略，该策略增加了在存储桶下使用任何对象/*的权限。

之前

...
"Resource": [
            "arn:aws:s3:::www.xyz.com"
            ]
...

之后

...
"Resource": [
            "arn:aws:s3:::www.xyz.com/*"
            ]
...