subprocess - Celery 任务子进程 stdout 记录

Question

我有一个 celery 任务，它使用 subprocess 调用 Django 应用程序外部的其他 python 脚本。该程序中有一些 print，我想将这些打印在我的 celery 日志文件或数据库中。当我在 Django settings.py 文件中设置 CELERY_ALWAYS_EAGER = True 时，一切正常。如果我不设置此设置，则 celery 任务仅在退出时记录子进程 ​​stdout。似乎 p.stdout.readline() 正在阻塞。

run-test.py 是一个很长的过程，只有几分钟，但它会打印出它正在做什么。我想捕捉这个。

@shared_task
def run_tests(scenario_path, vu):
    basedir = os.path.abspath(os.path.dirname(__file__))
    config_path = '%s/../../scripts/config.ini' % basedir
    cmd = ['python', '%s/../../scripts/aws/run-test.py' % basedir, '%s' % config_path, scenario_path, str(vu), str(2)]
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    while True:
        line = p.stdout.readline()
        if line != '':
            logger.info(line)
        else:
            return

Answer 1

我发现这非常有用，使用 select 进行轮询而不是在 readline 上阻塞。

https://gist.github.com/bgreenlee/1402841

child = subprocess.Popen(popenargs, stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE, **kwargs)

log_level = {child.stdout: stdout_log_level,
             child.stderr: stderr_log_level}

def check_io():
    ready_to_read = select.select([child.stdout, child.stderr], [], [], 1000)[0]
    for io in ready_to_read:
        line = io.readline()
        logger.log(log_level[io], line[:-1])

# keep checking stdout/stderr until the child exits
while child.poll() is None:
    check_io()

check_io()  # check again to catch anything after the process exits