google-app-engine - 将 Google Cloud KMS 与 Flask、gunicorn 和 gevent worker 一起使用时出错 : "google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded"

Question

在 Flask 请求的上下文中尝试使用 google-cloud-kms 进行加密或解密会导致在使用 gunicorn 和 gevent worker 提供 Flask 应用程序时出现以下错误。在本地运行应用程序时和在 Google App Engine(灵活)环境中运行应用程序时都会发生错误。

app_1  | Traceback (most recent call last):
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
app_1  |     return callable_(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
app_1  |     return _end_unary_response_blocking(state, call, False, None)
app_1  |   File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
app_1  |     raise _Rendezvous(state, None, None, deadline)
app_1  | grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
app_1  |    status = StatusCode.UNAVAILABLE
app_1  |    details = "Deadline Exceeded"
app_1  |    debug_error_string = "{"created":"@1561236057.820157100","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":14}"
app_1  | >
app_1  |
app_1  | The above exception was the direct cause of the following exception:
app_1  |
app_1  | Traceback (most recent call last):
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
app_1  |     response = self.full_dispatch_request()
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
app_1  |     rv = self.handle_user_exception(e)
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
app_1  |     reraise(exc_type, exc_value, tb)
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
app_1  |     raise value
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
app_1  |     rv = self.dispatch_request()
app_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
app_1  |     return self.view_functions[rule.endpoint](**req.view_args)
app_1  |   File "/app/app.py", line 14, in encrypt
app_1  |     encrypted = client.encrypt(key_name, plaintext.encode())
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/cloud/kms_v1/gapic/key_management_service_client.py", line 1286, in encrypt
app_1  |     request, retry=retry, timeout=timeout, metadata=metadata
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
app_1  |     return wrapped_func(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
app_1  |     on_error=on_error,
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
app_1  |     return target()
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
app_1  |     return func(*args, **kwargs)
app_1  |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
app_1  |     six.raise_from(exceptions.from_grpc_error(exc), exc)
app_1  |   File "<string>", line 3, in raise_from
app_1  | google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded

需求.txt

Flask==1.0.3
gevent==1.4.0
google-cloud-kms==1.0.0
grpcio==1.21.1
gunicorn==19.9.0

重现此内容的 Flask 应用示例:

from flask import Flask
from google.cloud import kms_v1
import base64


def create_app():
    app = Flask(__name__)

    @app.route('/')
    def encrypt():
        plaintext = "plain"
        client = kms_v1.KeyManagementServiceClient()
        key_name = client.crypto_key_path('project', 'location', 'keyring', 'key')
        encrypted = client.encrypt(key_name, plaintext.encode())
        return base64.urlsafe_b64encode(encrypted.ciphertext)

    return app


example_app = create_app()

服务于:

gunicorn -t 300 -b :8080 -k gevent -w 4 app:example_app

Answer 1

回答我自己的问题，因为我花了一些时间调试它，并在网上找到了类似但不是这个特定问题的答案。

问题似乎是 grpc 和 gevent 如何协同工作(或不协同工作)，最快的解决方法是使用另一个 gunicorn worker。经过测试，至少 sync、eventlet 和 gthread worker 可以防止发生此错误。

gunicorn -t 300 -b :8080 -k eventlet -w 4 app:example_app