python-3.x - 在 centos 7 上使用 systemd 守护 celery

Question

我打算使用 systemd 将 celery 4.3.0 作为守护进程运行，但它给了我这个错误:

它会启动 worker 但会很快停止它们。但是，我可以通过键入以下命令手动运行工作人员:

• celery -A bin.proj.celery_app worker1

celery.service - Celery Service
   Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fri 2020-04-10 08:39:59 CEST; 35s ago
  Process: 19276 ExecStop=/bin/sh -c ${CELERY_BIN} multi stopwait ${CELERYD_NODES} -- 
pidfile=${CELERYD_PID_FILE} (code=exited, status=0/SUCCESS)
  Process: 19245 ExecStart=/bin/sh -c ${CELERY_BIN} multi start ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} ${CELERYD_OPTS} (code=exited, status=0/SUCCESS)
 Main PID: 19245 (code=exited, status=0/SUCCESS)

Apr 10 08:39:58 prod1 sh[19276]: > Stopping nodes...
Apr 10 08:39:58 prod1 sh[19276]: > worker1@prod1: TERM -> 19254
Apr 10 08:39:58 prod1 sh[19276]: > worker2@prod1: TERM -> 19264
Apr 10 08:39:58 prod1 sh[19276]: > Waiting for 2 nodes -> 19254, 19264......
Apr 10 08:39:58 prod1 sh[19276]: > worker1@prod1: OK
Apr 10 08:39:58 prod1 sh[19276]: > worker1@prod1: DOWN
Apr 10 08:39:58 prod1 sh[19276]: > Waiting for 2 nodes -> None, None....
Apr 10 08:39:58 prod1 sh[19276]: > worker2@prod1: OK
Apr 10 08:39:58 prod1 sh[19276]: > worker2@prod1: DOWN
Apr 10 08:39:58 prod1 sh[19276]: > Waiting for 1 node -> None...

这是 celery.service:

[Unit]
Description=Celery Service
After=network.target

[Service]
type=forking
User=ssc
Group=ssc
EnvironmentFile=/etc/conf/celery
WorkingDirectory=/home/ssc/Mr.Robot/
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} restart ${CELERYD_NODES} -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

这是关联的配置文件:

# Name of nodes to start
# here we have a single node
#CELERYD_NODES="w1"
# or we could have three nodes:
CELERYD_NODES="worker1 worker2 worker3"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="bin.proj.celery_app"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--loglevel=INFO --time-limit=300 --concurrency=3"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"

和应用文件:

import sys,os
sys.path.append(os.getcwd())

from celery import Celery
from bin.proj.utils import create_rotating_log


# initialize logger
global logger
logger = create_rotating_log('./log/activity.log')

# initialize celery
try:
    app =   Celery('celery_app', 
                    broker="".join(['redis://:pass@localhost:6379/10']), 
                    backend="".join(['redis://:pass@localhost:6379/11']), 
                    include=['bin.proj.tasks'])
    #logger.info('Celery session OK')
except Exception as e:
    logger.error('Celery session KO')
    logger.error(e)

if __name__ == '__main__':
    app.start() 

用户是“ssc”，该用户不属于根组，我将项目所需的所有权限授予“ssc”用户。任何帮助将不胜感激，我已经为此花费了太多时间，谢谢

Answer 1

我在守护 celery 时遇到了类似的问题，问题与 pidfile 和 logfile 的权限有关。

目录/var/run/celery 和/var/log/celery 应该属于运行服务的用户，这里是用户scc。问题是当作为守护进程运行时，输出被关闭，从而隐藏了错误消息。

如果您想调试服务并找到问题，只需使用“scc”帐户运行完整命令即可。它应该会告诉您任何访问权限错误或其他任何信息。

/bin/sh -c '/usr/local/bin/celery multi start worker1 worker2 worker3 -A bin.proj.celery_app --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n%I.log --loglevel=INFO --time-limit=300 --concurrency=3'