ubuntu - supervisord 和 celery 热关机

Question

每当我在运行 Supervisor 和 Celery 的 ubuntu 实例上运行“sudo shutdown -h now”时，Celery 似乎只是在处理任务的过程中关闭，当查看 tail -f 时我没有看到“当 Celery 在前台运行时键入 ctrl-C 时，您通常会看到 warm shutdown"消息。

这对我来说是个大问题，因为服务器可能随时终止，因此等待 Celery 完成其任务的处理至关重要。这可能需要几秒钟、10 分钟或 10 小时，但关闭信号应该始终让 Celery 完成其工作。我在我的 celeryd 配置文件中添加了 stopasgroup 和 killasgroup，它似乎确实从主管停止 Celery 停止了所有子进程，但不知何故它似乎忽略了“sudo shutdown -h now”发送的关闭信号。

有谁知道为什么 Celery 没有显示通常的“热关机”？

这里是一个 worker conf 的例子:

[program:celery-worker]
command=/path/to/script-running-celery.sh
directory=/path/to/celery/project/tasks
user=myuser
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 60
killasgroup=true
stopasgroup=true
priority=50

下面是正在运行的实际 shell 脚本:

#!/bin/bash

export PATH="$PATH:/mylocal-envs/localenv";
cd /path/to/celery/project/tasks;
exec /mylocal-envs/localenv/bin/celery worker -c8 -n my_worker_%h_$(uuid) --without-gossip -A tasks --queues=myqueue --loglevel=INFO -Ofair

您能看出为什么关闭命令不会让 Celery 在热关闭状态下运行并结束其当前任务的原因吗？

编辑:请注意，当不使用 Supervisor 直接运行我的脚本并从另一个 ssh 窗口关闭时，我仍然得到相同的行为:工作人员正在处理任务，然后收到关闭信号并关闭 ssh session .以下是运行脚本时从 worker 前台发生的情况:

[2014-05-17 19:29:02,955: INFO/MainProcess] Task tasks.a_task[5143a89d-e077-4145-ba94-0932d0f81d9e] succeeded in 15.523131618s: u''
[2014-05-17 19:29:03,621: INFO/MainProcess] Received task: tasks.a_task[b61a30ba-2fbb-475a-9cb2-ec88d20a8885]

Broadcast message from jdoe@10.0.0.12
    (/dev/pts/1) at 19:29 ...

The system is going down for halt NOW!
[2014-05-17 19:29:06,846: INFO/MainProcess] Received task: tasks.a_task[9146f9f4-f06c-4563-b8d1-43c8d9367d9b]
[2014-05-17 19:29:08,089: INFO/MainProcess] Task tasks.a_task[b4c7dacc-3f6d-42e5-ab67-f33748f0af14] succeeded in 16.49714411s: u''
Connection to xx.xx.xxx.xx closed by remote host.
Connection to xx.xx.xxx.xx closed.

你仍然可以看到一个任务成功，另一个任务被接收。查看 flower，我看到一堆已启动的任务现在停留在“已启动”或“已接收”状态。

编辑:尝试直接在 command= 选项中运行 celery 而不是运行 shell 脚本后，我注意到相同的行为。如果我在前台通过 ssh 运行我的脚本(在 Debug模式下使用 celery)，我将在按下 ctrl-C 时得到以下信息:

worker: Hitting Ctrl+C again will terminate all running tasks!
289
290worker: Warm shutdown (MainProcess)
291[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Hub...
292[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Pool...
293[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Consumer...
294[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Worker: Stopping Consumer...
295[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Connection...
296[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Events...
297[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Mingle...
298[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Tasks...
299[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Control...
300[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Heart...
301[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing event loop...
302[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping event loop...
303[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping Heart...
304[2014-05-18 13:12:49,564: DEBUG/MainProcess] | Consumer: Stopping Control...
305[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Tasks...
306[2014-05-18 13:12:49,570: DEBUG/MainProcess] Cancelling task consumer...
307[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Mingle...
308[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Events...
309[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Consumer: Stopping Connection...
310[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Worker: Stopping Pool...
311[2014-05-18 13:12:50,608: DEBUG/MainProcess] | Worker: Stopping Hub...
312[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Heart...
313[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Control...
314[2014-05-18 13:12:50,613: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
315[2014-05-18 13:12:50,614: DEBUG/MainProcess] Cancelling task consumer...
316[2014-05-18 13:12:50,614: DEBUG/MainProcess] Closing consumer channel...
317[2014-05-18 13:12:50,615: DEBUG/MainProcess] | Consumer: Shutdown Events...
318[2014-05-18 13:12:50,616: DEBUG/MainProcess] | Consumer: Shutdown Connection...
319[2014-05-18 13:12:50,617: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

这看起来像是正常的关闭过程，我希望看到主管发生这种情况。另一方面，停止使用 supervisor 会给我以下结果:

[2014-05-18 13:14:28,369: DEBUG/MainProcess] pidbox received method ping() [reply_to:{u'routing_key': u'1bcaf3a6-0c77-33b9-9d46-56e8634e5b1d', u'exchange': u'reply.celery.pidbox'} ticket:161fcbdb-dac0-48c9-ac13-ab7a75bfdc85]

没有像您在按 ctrl-c 时看到的那样的关机跟踪。它只是停止了。

Answer 1

这似乎是 Ubuntu (3.0a) 附带的 Supervisor 版本中的错误。切换到当前稳定版本的 Supervisor 后(手动安装在/usr/local/bin 中)，我可以看到日志确实收到了 sigterm。