amazon-web-services - Celery 在 AWS Elastic Beanstalk 上启动时意外退出

Question

我在 Elastic Beanstalk 实例上启动 Celery 工作器时遇到问题。几秒钟后，它只是意外退出，没有错误。实例应该有足够的内存。我正在通过日志级别调试附加工作人员的输出(敏感信息由**代替)。任何指导都会非常有帮助。谢谢。

(venv) [ec2-user@ip-** app]$ celery -A app worker -l debug
[12/Mar/2018 10:18:29] DEBUG [raven.contrib.django.client.DjangoClient:265] Configuring Raven for host: <raven.conf.remote.RemoteConfig object at 0x7f556309e940>

 -------------- celery@ip-** v4.1.0 (latentcall)
---- **** ----- 
--- * ***  * -- Linux-4.9.75-25.55.amzn1.x86_64-x86_64-with-glibc2.3.4 2018-03-12 10:18:29
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         app:0x7f556998edd8
- ** ---------- .> transport:   sqs://**:**@localhost//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . common.tasks.send_templated_email
  . orders.tasks.import_orders_from_all_companies

[2018-03-12 10:18:29,783: DEBUG/MainProcess] Setting config variable for region to 'eu-central-1'
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable config_file from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable credentials_file from defaults.
[2018-03-12 10:18:29,784: DEBUG/MainProcess] Loading variable data_path from defaults.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable region from instance vars with value 'eu-central-1'.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,785: DEBUG/MainProcess] Loading variable ca_bundle from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading variable api_versions from defaults.
[2018-03-12 10:18:29,786: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/endpoints.json
[2018-03-12 10:18:29,790: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,790: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f55633fe048>
[2018-03-12 10:18:29,793: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2018-03-12 10:18:29,796: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f5563448d90>
[2018-03-12 10:18:29,797: DEBUG/MainProcess] The s3 config key is not a dictionary type, ignoring its value of: None
[2018-03-12 10:18:29,802: DEBUG/MainProcess] Setting sqs timeout as (60, 60)
[2018-03-12 10:18:29,802: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/_retry.json
[2018-03-12 10:18:29,803: DEBUG/MainProcess] Registering retry handlers for service: sqs
[2018-03-12 10:18:29,803: DEBUG/MainProcess] Event before-parameter-build.sqs.ListQueues: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Making request for OperationModel(name=ListQueues) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'ListQueues', 'Version': '2012-11-05', 'QueueNamePrefix': ''}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5562069f28>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Event request-created.sqs.ListQueues: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5562069cf8>>
[2018-03-12 10:18:29,804: DEBUG/MainProcess] Event choose-signer.sqs.ListQueues: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,805: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,805: DEBUG/MainProcess] CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z

content-type;host;x-amz-date
**
[2018-03-12 10:18:29,805: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,806: DEBUG/MainProcess] Signature:
**
[2018-03-12 10:18:29,806: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,807: INFO/MainProcess] Starting new HTTPS connection (1): eu-central-1.queue.amazonaws.com
[2018-03-12 10:18:29,839: DEBUG/MainProcess] "POST / HTTP/1.1" 200 409
[2018-03-12 10:18:29,840: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '409', 'connection': 'keep-alive', 'x-amzn-requestid': '**'}
[2018-03-12 10:18:29,840: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ListQueuesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ListQueuesResult><QueueUrl>**</QueueUrl><QueueUrl>**</QueueUrl></ListQueuesResult><ResponseMetadata><RequestId>**</RequestId></ResponseMetadata></ListQueuesResponse>'
[2018-03-12 10:18:29,841: DEBUG/MainProcess] Event needs-retry.sqs.ListQueues: calling handler <botocore.retryhandler.RetryHandler object at 0x7f556201c390>
[2018-03-12 10:18:29,841: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,841: INFO/MainProcess] Connected to sqs://**:**@localhost//
[2018-03-12 10:18:29,850: DEBUG/MainProcess] Setting config variable for region to 'eu-central-1'
[2018-03-12 10:18:29,850: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable config_file from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable credentials_file from defaults.
[2018-03-12 10:18:29,851: DEBUG/MainProcess] Loading variable data_path from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable region from instance vars with value 'eu-central-1'.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable ca_bundle from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,852: DEBUG/MainProcess] Loading variable api_versions from defaults.
[2018-03-12 10:18:29,853: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/endpoints.json
[2018-03-12 10:18:29,857: DEBUG/MainProcess] Loading variable profile from defaults.
[2018-03-12 10:18:29,857: DEBUG/MainProcess] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f55633fe048>
[2018-03-12 10:18:29,861: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/sqs/2012-11-05/service-2.json
[2018-03-12 10:18:29,863: DEBUG/MainProcess] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f5563448d90>
[2018-03-12 10:18:29,863: DEBUG/MainProcess] The s3 config key is not a dictionary type, ignoring its value of: None
[2018-03-12 10:18:29,865: DEBUG/MainProcess] Setting sqs timeout as (60, 60)
[2018-03-12 10:18:29,865: DEBUG/MainProcess] Loading JSON file: /opt/python/run/venv/local/lib/python3.6/site-packages/botocore/data/_retry.json
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Registering retry handlers for service: sqs
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Event before-parameter-build.sqs.ListQueues: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,866: DEBUG/MainProcess] Making request for OperationModel(name=ListQueues) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'ListQueues', 'Version': '2012-11-05', 'QueueNamePrefix': ''}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5561acfbe0>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Event request-created.sqs.ListQueues: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5561acfb38>>
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Event choose-signer.sqs.ListQueues: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,867: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,868: DEBUG/MainProcess] CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z

content-type;host;x-amz-date
**
[2018-03-12 10:18:29,868: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,868: DEBUG/MainProcess] Signature:
**
[2018-03-12 10:18:29,869: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,869: INFO/MainProcess] Starting new HTTPS connection (1): eu-central-1.queue.amazonaws.com
[2018-03-12 10:18:29,895: DEBUG/MainProcess] "POST / HTTP/1.1" 200 409
[2018-03-12 10:18:29,895: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '409', 'connection': 'keep-alive', 'x-amzn-requestid': '5b132733-22e0-5567-8762-74136ac526ec'}
[2018-03-12 10:18:29,896: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ListQueuesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ListQueuesResult><QueueUrl>**</QueueUrl><QueueUrl>**</QueueUrl></ListQueuesResult><ResponseMetadata><RequestId>5b132733-22e0-5567-8762-74136ac526ec</RequestId></ResponseMetadata></ListQueuesResponse>'
[2018-03-12 10:18:29,896: DEBUG/MainProcess] Event needs-retry.sqs.ListQueues: calling handler <botocore.retryhandler.RetryHandler object at 0x7f5561a763c8>
[2018-03-12 10:18:29,896: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,899: DEBUG/MainProcess] Event before-parameter-build.sqs.GetQueueAttributes: calling handler <function generate_idempotent_uuid at 0x7f5563406378>
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Making request for OperationModel(name=GetQueueAttributes) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.5.16 Python/3.6.2 Linux/4.9.75-25.55.amzn1.x86_64 Botocore/1.8.30'}, 'body': {'Action': 'GetQueueAttributes', 'Version': '2012-11-05', 'QueueUrl': '**', 'AttributeName.1': 'ApproximateNumberOfMessages'}, 'url': 'https://eu-central-1.queue.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f5562069f28>, 'has_streaming_input': False, 'auth_type': None}}
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Event request-created.sqs.GetQueueAttributes: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5562069cf8>>
[2018-03-12 10:18:29,900: DEBUG/MainProcess] Event choose-signer.sqs.GetQueueAttributes: calling handler <function set_operation_specific_signer at 0x7f5563406268>
[2018-03-12 10:18:29,901: DEBUG/MainProcess] Calculating signature using v4 auth.
[2018-03-12 10:18:29,901: DEBUG/MainProcess] CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:eu-central-1.queue.amazonaws.com
x-amz-date:20180312T091829Z

content-type;host;x-amz-date
**
[2018-03-12 10:18:29,901: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20180312T091829Z
20180312/eu-central-1/sqs/aws4_request
**
[2018-03-12 10:18:29,901: DEBUG/MainProcess] Signature:
9fb0d1ad68b5d25bf148cc11857b1e1083418557229ca2c47e8b525b54880b74
[2018-03-12 10:18:29,902: DEBUG/MainProcess] Sending http request: <PreparedRequest [POST]>
[2018-03-12 10:18:29,910: DEBUG/MainProcess] "POST / HTTP/1.1" 200 357
[2018-03-12 10:18:29,911: DEBUG/MainProcess] Response headers: {'server': 'Server', 'date': 'Mon, 12 Mar 2018 09:18:29 GMT', 'content-type': 'text/xml', 'content-length': '357', 'connection': 'keep-alive', 'x-amzn-requestid': '9aa38f1d-25e5-576d-a9a9-dc3d6dc029a0'}
[2018-03-12 10:18:29,911: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><GetQueueAttributesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><GetQueueAttributesResult><Attribute><Name>ApproximateNumberOfMessages</Name><Value>0</Value></Attribute></GetQueueAttributesResult><ResponseMetadata><RequestId>**</RequestId></ResponseMetadata></GetQueueAttributesResponse>'
[2018-03-12 10:18:29,912: DEBUG/MainProcess] Event needs-retry.sqs.GetQueueAttributes: calling handler <botocore.retryhandler.RetryHandler object at 0x7f556201c390>
[2018-03-12 10:18:29,912: DEBUG/MainProcess] No retry needed.
[2018-03-12 10:18:29,921: DEBUG/MainProcess] Canceling task consumer...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] Canceling task consumer...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] Closing consumer channel...
[2018-03-12 10:18:30,926: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

Answer 1

我已经解决了这个问题。 Amazon 实例需要 PyCurl 和一些额外的包才能与 SQS 正确连接。

我建议你已经有一个脚本文件来运行 celery daemon( run_supervised_celeryd.sh )

这是我的 EB 配置文件:

packages:
  yum:
    libjpeg-turbo-devel: []
    libpng-devel: []
    libcurl-devel: []


container_commands:
  01_migrate:
    command: "django-admin.py migrate --noinput"
    leader_only: true
  02_collectstatic:
    command: "python manage.py collectstatic --noinput"
  03_pycurl:
    command: 'source /opt/python/run/venv/bin/activate && pip3 install /usr/local/share/pycurl-7.43.0.tar.gz --global-option="--with-nss" --upgrade'
  04_celery_tasks_run:
    command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
    leader_only: true

files:
  "/usr/local/share/pycurl-7.43.0.tar.gz" :
    mode: "000644"
    owner: root
    group: root
    source: https://pypi.python.org/packages/source/p/pycurl/pycurl-7.43.0.tar.gz

添加环境变量 PYCURL_SSL_LIBRARY="nss"

所有列出的设置都解决了问题