python - 大量的multiprocessing.Process导致死锁

Question

上下文

我需要在 multiprocessing.ThreadPool 中运行 multiprocessing.Process。起初看起来很奇怪，但这是我发现处理可能发生的段错误的唯一方法，因为我使用的是 C++ 共享库。如果附加了段错误，进程将被终止，我可以检查 process.exitcode 并进行处理。

问题

一段时间后，当我尝试加入进程时出现死锁。

这是我的代码的一个简单版本:

import sys, time, multiprocessing
from multiprocessing.pool import ThreadPool

def main():
    # Launch 8 workers
    pool = ThreadPool(8)
    it = pool.imap(run, range(500))
    while True:
        try:
            it.next()
        except StopIteration:
            break

def run(value):
    # Each worker launch it own Process
    process = multiprocessing.Process(target=run_and_might_segfault,     args=(value,))
    process.start()

    while process.is_alive():
        sys.stdout.write('.')
        sys.stdout.flush()
        time.sleep(0.1)

    # Will never join after a while, because of a mystery deadlock
    process.join()

    # Deals with process.exitcode to log errors

def run_and_might_segfault(value):
    # Load a shared library and do stuff (could throw c++ exception, segfault ...)
    print(value)

if __name__ == '__main__':
    main()

这是一个可能的输出:

➜  ~ python m.py
..0
1
........8
.9
.......10
......11
........12
13
........14
........16
........................................................................................

如您所见，process.is_alive() 在几次迭代后始终为真，进程永远不会加入。

如果我 CTRL-C 脚本 a 得到这个堆栈跟踪:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 680, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "m.py", line 30, in <module>
    main()
  File "m.py", line 9, in main
    it.next()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5    /lib/python3.5/multiprocessing/pool.py", line 684, in next
    self._cond.wait(timeout)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5    /lib/python3.5/threading.py", line 293, in wait
    waiter.acquire()
KeyboardInterrupt

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5    /lib/python3.5/multiprocessing/popen_fork.py", line 29, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

附言在 macOS 上使用 python 3.5.2。

感谢各种帮助，谢谢。

编辑

我尝试使用 python 2.7，它运行良好。可能只是 python 3.5 的问题？

Answer 1

该问题也在最新版本的 CPython 上重现 - Python 3.7.0a0(默认值:4e2cce65e522，2016 年 10 月 13 日，21:55:44)。

如果你attach对于 gdb 的卡住进程之一，您会看到它正试图在 sys.stdout.flush() 调用中获取锁:

(gdb) py-list
 263                import traceback
 264                sys.stderr.write('Process %s:\n' % self.name)
 265                traceback.print_exc()
 266            finally:
 267                util.info('process exiting with exitcode %d' % exitcode)
>268                sys.stdout.flush()
 269                sys.stderr.flush()
 270
 271            return exitcode

Python 级别的回溯看起来像这样:

 (gdb) py-bt
 Traceback (most recent call first):
   File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/process.py", line 268, in _bootstrap
     sys.stdout.flush()
   File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/popen_fork.py", line 74, in _launch
     code = process_obj._bootstrap()
   File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/popen_fork.py", line 20, in __init__
     self._launch(process_obj)
   File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/context.py", line 277, in _Popen
     return Popen(process_obj)
   File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/context.py", line 223, in _Popen
     return _default_context.get_context().Process._Popen(process_obj)
   File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/process.py", line 105, in start
     self._popen = self._Popen(self)
   File "deadlock.py", line 17, in run
     process.start()
   File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/pool.py", line 119, in worker
     result = (True, func(*args, **kwds))
   File "/home/rpodolyaka/src/cpython/Lib/threading.py", line 864, in run
     self._target(*self._args, **self._kwargs)
   File "/home/rpodolyaka/src/cpython/Lib/threading.py", line 916, in _bootstrap_inner
     self.run()
   File "/home/rpodolyaka/src/cpython/Lib/threading.py", line 884, in _bootstrap
     self._bootstrap_inner()

在解释器级别它看起来像:

(gdb) frame 6

(gdb) list
287        return 0;
288    }
289    relax_locking = (_Py_Finalizing != NULL);
290    Py_BEGIN_ALLOW_THREADS
291    if (!relax_locking)
292        st = PyThread_acquire_lock(self->lock, 1);
293    else {
294        /* When finalizing, we don't want a deadlock to happen with daemon
295         * threads abruptly shut down while they owned the lock.
296         * Therefore, only wait for a grace period (1 s.). ... */

(gdb) p /x self->lock
$1 = 0xd25ce0

(gdb) p /x self->owner
$2 = 0x7f9bb2128700

请注意，从这个特定子进程的角度来看，锁仍然由父进程中的一个线程拥有(LWP 1105):

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7f9bb5559440 (LWP 1102) "python" 0x00007f9bb5157577 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0xe4d340) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  2    Thread 0x7f9bb312a700 (LWP 1103) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  3    Thread 0x7f9bb2929700 (LWP 1104) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  4    Thread 0x7f9bb2128700 (LWP 1105) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  5    Thread 0x7f9bb1927700 (LWP 1106) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  6    Thread 0x7f9bb1126700 (LWP 1107) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  7    Thread 0x7f9bb0925700 (LWP 1108) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  8    Thread 0x7f9b9bfff700 (LWP 1109) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  9    Thread 0x7f9b9b7fe700 (LWP 1110) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  10   Thread 0x7f9b9affd700 (LWP 1111) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
  11   Thread 0x7f9b9a7fc700 (LWP 1112) "python" 0x00007f9bb5157577 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x7f9b80001ed0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  12   Thread 0x7f9b99ffb700 (LWP 1113) "python" 0x00007f9bb5157577 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x7f9b84001bb0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205

所以这确实是一个死锁，它的发生是由于您在多个 sys.stdout 上执行写入和刷新在原始进程中同时创建多个线程，同时还创建子进程 - 根据 fork(2) 系统调用的性质子进程继承父内存，包括获取的锁:fork() 调用必须在获取锁时执行，即使父进程最终释放它，子进程也看不到，因为每个它们中的一个现在有自己的内存空间，在写入时被复制。

因此，在混合时需要非常小心多线程与多处理并确保所有锁在 fork() 之前正确释放，如果他们要在子进程中使用。

它与 http://bugs.python.org/issue6721 中描述的非常相似

请注意，如果您从代码段中删除与 sys.stdout 的交互，它将正常工作。