python - Celery:将每个任务运行记录到它自己的文件中？

Question

我希望每个运行的作业都记录到 logs/目录中它自己的文件，其中文件名是 taskid。

logger = get_task_logger(__name__)

@app.task(base=CallbackTask)
def calc(syntax):
    some_func()
    logger.info('started')

在我的 worker 中，我使用 -f 参数设置要输出的日志文件。我想确保它将每个任务输出到它自己的日志文件中。

Answer 1

好像我迟到了 3 年。不过，这是我的解决方案，灵感来自@Mikko Ohtamaa 的想法。我只是通过使用 Celery Signals 和 python 的内置日志记录框架来准备和清理日志记录句柄，使其略有不同。

from celery.signals import task_prerun, task_postrun
import logging

# to control the tasks that required logging mechanism
TASK_WITH_LOGGING = ['Proj.tasks.calc']
@task_prerun.connect(sender=TASK_WITH_LOGGING)
def prepare_logging(signal=None, sender=None, task_id=None, task=None, args=None, kwargs=None):
    logger = logging.getLogger(task_id)
    formatter = logging.Formatter('[%(asctime)s][%(levelname)s] %(message)s')
    # optionally logging on the Console as well as file
    stream_handler = logging.StreamHandler()
    stream_handler.setFormatter(formatter)
    stream_handler.setLevel(logging.INFO)
    # Adding File Handle with file path. Filename is task_id
    task_handler = logging.FileHandler(os.path.join('/tmp/', task_id+'.log'))
    task_handler.setFormatter(formatter)
    task_handler.setLevel(logging.INFO)
    logger.addHandler(stream_handler)
    logger.addHandler(task_handler)

@task_postrun.connect(sender=TASK_WITH_LOGGING)
def close_logging(signal=None, sender=None, task_id=None, task=None, args=None, kwargs=None, retval=None, state=None):
    # getting the same logger and closing all handles associated with it
    logger = logging.getLogger(task_id)
    for handler in logger.handlers:
        handler.flush()
        handler.close()
    logger.handlers = []

@app.task(base=CallbackTask, bind=True)
def calc(self, syntax):
    # getting logger with name Task ID. This is already
    # created and setup in prepare_logging
    logger = logging.getLogger(self.request.id)
    some_func()
    logger.info('started')

bind=True在这里是必要的，以便在任务中使用 id。这将使用 <task_id>.log 创建单独的日志文件每次任务calc被执行。