python - celery 错过了心跳(on_node_lost)

Question

我刚刚升级到 celery 3.1，现在我在我的日志中看到了这个::

on_node_lost - INFO - missed heartbeat from celery@queue_name for every queue/worker in my cluster.

根据文档，BROKER_HEARTBEAT 默认关闭，我还没有配置它。

我应该明确设置 BROKER_HEARTBEAT=0 还是应该检查其他内容？

Answer 1

Celery 3.1 添加了新的 mingle 和 gossip 程序。我也得到了很多错过的心跳和传递 --without-gossip 给我的 worker 清除了它。

https://docs.celeryproject.org/en/3.1/whatsnew-3.1.html#mingle-worker-synchronization

Mingle: Worker synchronization

The worker will now attempt to synchronize with other workers in thesame cluster.

Synchronized data currently includes revoked tasks and logical clock.

This only happens at startup and causes a one second startup delay tocollect broadcast responses from other workers.

You can disable this bootstep using the --without-mingle argument.

https://docs.celeryproject.org/en/3.1/whatsnew-3.1.html#gossip-worker-worker-communication

Gossip: Worker <-> Worker communication

Workers are now passively subscribing to worker related events likeheartbeats.

This means that a worker knows what other workers are doing and candetect if they go offline. Currently this is only used for clocksynchronization, but there are many possibilities for future additionsand you can write extensions that take advantage of this already.

Some ideas include consensus protocols, reroute task to best worker(based on resource usage or data locality) or restarting workers whenthey crash.

We believe that although this is a small addition, it opens amazingpossibilities.

You can disable this bootstep using the --without-gossip argument.