django - 如何检测陈旧的工作人员(或自动重启)

Question

我们最近遇到了 celery 框架的糟糕情况。队列中有很多消息，但未处理这些消息。我们重新启动了 celery ，消息又开始被处理了。但是，我们不希望这种情况再次发生，并且正在寻找永久解决方案。

看起来 celery 的 worker 已经过时了。 celery 的文档记录了以下关于陈旧 worker 的信息:

This shows that there’s 2891 messages waiting to be processed in the task queue, and there are two consumers processing them.

One reason that the queue is never emptied could be that you have a stale worker process taking the messages hostage. This could happen if the worker wasn’t properly shut down.

When a message is received by a worker the broker waits for it to be acknowledged before marking the message as processed. The broker will not re-send that message to another consumer until the consumer is shut down properly.

If you hit this problem you have to kill all workers manually and restart them

See documentation

然而，这依赖于人工检查过时的工作人员，留下了很大的错误空间和人工成本。保持 celery 正常工作的好的解决方案是什么？

Answer 1

您可以使用主管或类似主管的工具来部署工作人员，请参阅 Running the worker as daemon .

此外，您可以使用 rabbitmq-management 监控队列状态, 要检查队列是否变得太大，假设您使用的是 RabbitMQ； celery monitoring还提供了一些监控机制