django - Celery 立即超出了 Heroku 上的内存

Question

我正在 Heroku 上部署一个 Celery 进程，每次启动时，它都会立即开始增加内存使用量，并在超过最大值后崩溃。

我只有一个名为“test_task”的任务，每分钟打印一次。

这是使用 Celery 的 Django 应用程序，后端托管在 Heroku 上的 Redis。

过程文件:

web: daphne chatbot.asgi:channel_layer --port $PORT --bind 0.0.0.0 --verbosity 1
chatworker: python manage.py runworker --verbosity 1
celeryworker: celery -A chatbot worker -l info

Heroku 日志:

app[celeryworker.1]: [INFO] 2018-09-11 23:06:36,710 : Scheduling celery jobs...
app[celeryworker.1]: [INFO] 2018-09-11 23:06:36,880 : adding minute task
app[celeryworker.1]: 
app[celeryworker.1]: -------------- celery@f46a12fb-1666-449a-b287-2ce90f95cf2c v4.2.0 (windowlicker)
app[celeryworker.1]: --- * ***  * -- Linux-4.4.0-1027-aws-x86_64-with-debian-jessie-sid 2018-09-11 23:06:37
app[celeryworker.1]: -- * - **** ---
app[celeryworker.1]: ---- **** -----
app[celeryworker.1]: - ** ---------- [config]
app[celeryworker.1]: - ** ---------- .> app:         chatbot:0x7f1c2a5ad5d0
app[celeryworker.1]: - ** ---------- .> transport:   redis://east-1-4.ec2.cloud.redislabs.com:13275//
app[celeryworker.1]: - ** ---------- .> results:     disabled://
app[celeryworker.1]: - *** --- * --- .> concurrency: 8 (prefork)
app[celeryworker.1]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
app[celeryworker.1]: --- ***** -----
app[celeryworker.1]: -------------- [queues]
app[celeryworker.1]: .> celery           exchange=celery(direct) key=celery
app[celeryworker.1]: 
app[celeryworker.1]: 
app[celeryworker.1]: [tasks]
app[celeryworker.1]: . companies.tasks.test_task
app[celeryworker.1]: 
heroku[celeryworker.1]: Process running mem=616M(120.4%)
heroku[celeryworker.1]: Error R14 (Memory quota exceeded)
app[api]: Build succeeded
app[celeryworker.1]: [2018-09-11 23:06:55,033: INFO/MainProcess] Connected to redis://rediscloud:**@redis-13275.c12.us-east-1-4.ec2.cloud.redislabs.com:13275//
app[celeryworker.1]: [2018-09-11 23:06:55,201: INFO/MainProcess] mingle: searching for neighbors
app[celeryworker.1]: [2018-09-11 23:06:56,462: INFO/MainProcess] mingle: all alone
heroku[celeryworker.1]: Process running mem=1063M(207.8%)
heroku[celeryworker.1]: Error R15 (Memory quota vastly exceeded)
heroku[celeryworker.1]: Stopping process with SIGKILL
heroku[celeryworker.1]: State changed from up to crashed
heroku[celeryworker.1]: State changed from crashed to starting
heroku[celeryworker.1]: Process exited with status 137

我明显做错了什么吗？

Answer 1

事实证明，将 celery 工作进程中的 --concurrency 参数更改为较低的数字是有效的。

我将其设置为celery -A chatbot worker -l info --concurrency 2，它似乎正在工作!