python - botocore s3 put 由于编码而出现哈希文件问题？

Question

我无法弄清楚为什么加载了编码 utf-8 的文件(其内容为“稍后删除我”)在散列时会在 botocore 中导致异常。

with io.open('deleteme','r', encoding='utf-8') as f:
  try:
    resp=client.put_object(
    Body=f,
    Bucket='s3-bucket-actual-name-for-real',
    Key='testing/a/put'
    )
    print('deleteme exists')
    print(resp)
  except:
    print('deleteme could not put')
    raise

产生:

deleteme could not put
Traceback (most recent call last): File
"./test_operator.py", line 41, in
Key='testing/a/put' File "/Users/lamblin/VEnvs/awscli/lib/python3.6/site-packages/botocore/client.py",
line 312, in _api_call
return self._make_api_call(operation_name, kwargs) File "/Users/lamblin/VEnvs/awscli/lib/python3.6/site-packages/botocore/client.py",
line 582, in _make_api_call request_signer=self._request_signer, context=request_context) File
"/Users/lamblin/VEnvs/awscli/lib/python3.6/site-packages/botocore/hooks.py",
line 242, in emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True) File
"/Users/lamblin/VEnvs/awscli/lib/python3.6/site-packages/botocore/hooks.py",
line 210, in _emit
response = handler(**kwargs) File "/Users/lamblin/VEnvs/awscli/lib/python3.6/site-packages/botocore/handlers.py",
line 201, in conditionally_calculate_md5
calculate_md5(params, **kwargs) File "/Users/lamblin/VEnvs/awscli/lib/python3.6/site-packages/botocore/handlers.py",
line 179, in calculate_md5
binary_md5 = _calculate_md5_from_file(body) File "/Users/lamblin/VEnvs/awscli/lib/python3.6/site-packages/botocore/handlers.py",
line 193, in _calculate_md5_from_file md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

现在可以通过使用 'rb' 打开文件来避免这种情况，但是文件对象 f 显然使用了编码吗？

Answer 1

Now this can be avoided by opening the file with 'rb' but, isn't the file object f clearly using an encoding?

mode='r' 中指定给 io.open 的编码用于解码 内容。所以当你迭代f时，内容已经被Python从bytes转换为str(文本)。

要直接与 botocore 交互，请使用 'rb' 模式打开文件，并删除编码 kwarg。当 botocore 为了传输内容而必须做的第一件事只是再次编码回字节时，没有必要将它解码为文本。