python - 谷歌云存储 api 性能回归

Question

我有一个 python 3.5 应用程序，它使用 python sdk 调用 Google Cloud Storage .

每隔一段时间，在 10-30 分钟内，对 API 的所有调用都会因 BrokenPipeError 或 ssl.SSLError 错误而失败。一段时间后，它们又开始工作了，我没有注意到原因的模式。

这是一个已知问题吗？它是特定于 python sdk 还是这是 google 方面的真正性能回归？

还应注意，这些错误将源自在我的本地计算机和 GCE 计算机上运行的相同代码。

BrokenPipe 的跟踪:

Traceback (most recent call last):
  File "oauth2client/util.py", line 140, in positional_wrapper
              return wrapped(*args, **kwargs)
  File "googleapiclient/http.py", line 722, in execute
                                     body=self.body, headers=self.headers)
  File "oauth2client/client.py", line 596, in new_request
                                           redirections, connection_type)
  File "httplib2/__init__.py", line 1314, in request
                      (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "httplib2/__init__.py", line 1064, in _request
          (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "httplib2/__init__.py", line 988, in _conn_request
                  conn.request(method, request_uri, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
          self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1128, in _send_request
          self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1079, in endheaders
          self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 911, in _send_output
          self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 885, in send
              self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
                  v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
              return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
          return self._sslobj.write(data)
BrokenPipeError[Errno 32] Broken pipe

ssl.SSLError 的跟踪:

  File "oauth2client/util.py", line 140, in positional_wrapper
              return wrapped(*args, **kwargs)
  File "googleapiclient/http.py", line 722, in execute
                                     body=self.body, headers=self.headers)
  File "oauth2client/client.py", line 596, in new_request
                                           redirections, connection_type)
  File "httplib2/__init__.py", line 1314, in request
                      (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "httplib2/__init__.py", line 1064, in _request
          (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "httplib2/__init__.py", line 1017, in _conn_request
                  response = conn.getresponse()
  File "/usr/lib/python3.5/http/client.py", line 1174, in getresponse
                  response.begin()
  File "/usr/lib/python3.5/http/client.py", line 282, in begin
              version, status, reason = self._read_status()
  File "/usr/lib/python3.5/http/client.py", line 243, in _read_status
          line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
                  return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 924, in recv_into
              return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 786, in read
              return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 570, in read
              v = self._sslobj.read(len, buffer)
ssl.SSLError[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1974)

Answer 1

绝对看起来像是 Google 方面的间歇性问题。破损的管道问题与 httpclient2 无法将现有连接重新连接到其 API 相关，这是对我们的服务影响最大的错误。我们也偶尔会收到“503 后端错误”。

我们的“解决方案”基本上是通过在完成后释放客户端并为下一个请求创建一个新客户端来允许连接自行关闭。

请记住，虽然我们的请求非常稀少，但使用云存储作为主要存储的服务可能希望尽可能长时间地保持连接打开。