python - python 线程中的多处理并不能让每个子进程都工作

Question

当在 python 中使用 threading 模块在每个线程中执行子进程时，某些进程无法正常启动并挂起，就像示例代码的输出一样。

由于 python 中的 IPC，启动进程似乎需要独占控制。是吗？

使用线程锁，效果完美。我只是想创建一个线程来控制子进程，进行其生死管理。

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# vim:fenc=utf-8

import multiprocessing
import threading

import fire
from logzero import logger


def process_method():
    logger.info("process log")


def start_process():
    logger.info("starting thread")
    process = multiprocessing.Process(target=process_method)
    process.daemon = True
    process.start()
    process.join()


def main(num_of_threads=3):
    threads = []
    for _ in range(num_of_threads):
        t = threading.Thread(target=start_process)
        t.daemon = True
        t.start()
        threads.append(t)
    for t in threads:
        t.join()
    logger.info("program done")


if __name__ == "__main__":
    fire.Fire(main)

输出

$ python '/tmp/tmp.lY3YDIltYg/test.py' 30
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:20] starting thread
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log
[I 190822 09:10:38 test:16] process log

环境

• Python 3.6
• ubutu 64 位

Answer 1

我假设您使用的是类 UNIX 系统？因为如果是这样，你就在这里做坏事； mixing fork with threads is a bad idea ，以及 Python 的默认实现 Process在类 UNIX 系统上使用 fork ，所以从 Process 开始es 在一个线程中，你是 fork来自多线程进程。

这里有两种解决方案:

1. 停止使用线程。这里不需要你的线程，因为它们实际上并不管理生命周期(线程和进程都是 daemon ，所以当主进程的主线程完成时它们会被毫不客气地杀死)。
2. 如果您必须使用线程，并且您使用的是 Python 3.4+，则可以切换 to the fork-server start method (或者使用 'spawn' 方法使您的代码可移植到 Windows，但代价是使 Process 在类 UNIX 系统上创建速度稍慢)，因此唯一的 fork从你的主进程开始是在你启动任何线程之前完成的，以及所有 future 的fork是从(无线程)fork 服务器完成的。您所要做的就是添加 multiprocessing.set_start_method('forkserver')作为 if __name__ == "__main__": 之后的第一行守卫。