python - GCloud 上传 httplib2.RedirectMissingLocation : Redirected but the response is missing a Location: header

Question

我正在尝试使用一个简单的 python 程序将一个小文件上传到 gcloud

client = storage.Client(project=GCLOUD_PROJECT)
bucket = client.get_bucket(GCLOUD_BUCKET)
blob = bucket.blob(GCLOUD_FILE_ON_CLOUD)
blob.upload_from_filename(GCLOUD_FILE_LOCAL)

它一直工作到最近，并且发生了一些变化。现在，每当我上传大于 5MB 的文件时，都会出现以下错误。小于或等于 5MB 的文件通过。大小不足以实现可续传，是吗？

Traceback (most recent call last):
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/gcloud_upload.py", line 40, in <module>
    blob.upload_from_filename(GCLOUD_FILE_LOCAL)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 597, in upload_from_filename
    encryption_key=encryption_key, client=client)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 543, in upload_from_file
    http_response = upload.stream_file(use_chunks=True)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1086, in stream_file
    response = send_func(self.stream.tell())
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1215, in _send_chunk
    return self._send_media_request(request, end)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1125, in _send_media_request
    self.bytes_http, request, retries=self.num_retries)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 423, in make_api_request
    check_response_func=check_response_func)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 371, in _make_api_request_no_retry
    redirections=redirections, connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1986, in request
    cachekey,
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1685, in _request
    content,
httplib2.RedirectMissingLocation: Redirected but the response is missing a Location: header.

当我调试时，我看到以下内容。

{
 'content-type': 'text/plain; charset=utf-8', 
 'range': 'bytes=0-1048575', 
 'content-length': '0', 
 'date': 'Sun, 19 Jan 2020 23:52:13 GMT', 
 'server': 'UploadServer', 
 'alt-svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000', 
 'status': '308'}

我的 pip 列表供引用。操作系统是 MAC OSx。我也在 Linux 上测试过。同样的问题。

adal                     1.2.2    
bson                     0.5.8    
cachetools               3.1.1    
certifi                  2019.9.11
cffi                     1.13.1   
chardet                  3.0.4    
cryptography             2.8      
dnspython                1.16.0   
gcloud                   0.18.3   
gitdb2                   2.0.6    
GitPython                3.0.5    
google-auth              1.6.3    
googleapis-common-protos 1.51.0   
httplib2                 0.16.0   
idna                     2.7      
itsdangerous             1.1.0    
MarkupSafe               1.1.1    
oauth2client             4.1.3    
oauthlib                 3.1.0    
pip                      19.0.3   
protobuf                 3.11.2   
pyasn1                   0.4.7    
pyasn1-modules           0.2.7    
pycparser                2.19     
PyJWT                    1.7.1    
python-dateutil          2.8.0    
requests                 2.19.1   
requests-oauthlib        1.2.0    
rsa                      4.0      
setuptools               40.8.0   
six                      1.12.0   
smmap2                   2.0.5    
urllib3                  1.23     
websocket-client         0.56.0   
Werkzeug                 0.16.0   

这是上周的工作。最近有什么变化吗？

Answer 1

解决方案
gcloud包被弃用两次并且与 httplib2>=0.16 不兼容.正确的解决方法是使用 google-cloud-*包家人。
google-api-python-client>=1.7.12正在使用redirect_codes API，请升级，它可以正常工作。

httplib2 v0.17.0 刚刚发布，能够修改被视为重定向的响应代码集。如果您可以修改创建 Http 的代码，这是最好的选择。目的:

http = httplib2.Http()
http.redirect_codes = http.redirect_codes - {308}

如果这是不可能的，请编辑您的 requirements.txt 以锁定 httplib2<0.16.0
很长的故事

google cloud storage server uses HTTP 308专用 resumable uploads feature这有点类似于“在同一位置重试相同的方法”，但不完全相同。

以上是(可能)PyPI 包的基本原理 google-resumable-media与 gcloud 相关软件包的更新版本使用，并以类似方式处理 200 和 308，与通用 HTTP 客户端不同。

历史背景:

2016 package gcloud已弃用，支持包 google-cloud

2017 年 google-cloud-python switches HTTP transport从 httplib2 到请求

2018 package google-cloud再次弃用 google-cloud-* 软件包

2020 httplib2 v0.16 获得了对每个 RFC7538 308 个重定向的支持

对不起，坏消息。作为 HTTP 爱好者，我偏向于 308 支持。如果您有更好的想法如何更优雅地处理这种情况，请联系。