python multiprocessing .join() 死锁取决于工作函数

Question

我正在使用 multiprocessing生成 4 的 python 库 Process()对象并行化 cpu 密集型任务。任务(来自这个伟大的 article 的灵感和代码)是计算列表中每个整数的质因数。

ma​​in.py:

import random
import multiprocessing
import sys

num_inputs  = 4000
num_procs   = 4
proc_inputs = num_inputs/num_procs
input_list  = [int(1000*random.random()) for i in xrange(num_inputs)]

output_queue = multiprocessing.Queue()
procs        = []
for p_i in xrange(num_procs):
  print "Process [%d]"%p_i
  proc_list = input_list[proc_inputs * p_i:proc_inputs * (p_i + 1)]
  print " - num inputs: [%d]"%len(proc_list)

  # Using target=worker1 HANGS on join
  p = multiprocessing.Process(target=worker1, args=(p_i, proc_list, output_queue))
  # Using target=worker2 RETURNS with success
  #p = multiprocessing.Process(target=worker2, args=(p_i, proc_list, output_queue))

  procs.append(p)
  p.start()

for p in jobs:
  print "joining ", p, output_queue.qsize(), output_queue.full()
  p.join()
  print "joined  ", p, output_queue.qsize(), output_queue.full()

print "Processing complete."
ret_vals = []
while output_queue.empty() == False:
    ret_vals.append(output_queue.get())
print len(ret_vals)
print sys.getsizeof(ret_vals)

观察:

• 如果每个进程的目标是函数 worker1，对于大于 4000 个元素的输入列表，主线程会卡在 .join() 上，等待生成的进程终止并且永不返回。
• 如果每个进程的目标是函数 worker2，对于相同的输入列表，代码工作正常并且主线程返回。

这让我很困惑，因为 worker1 和 worker2 之间的唯一区别(见下文)是前者在 Queue 而后者为每个进程插入一个列表列表。

为什么使用 worker1 而没有使用 worker2 目标会出现死锁？不应该两者(或两者都不)超出 Multiprocessing Queue maxsize limit is 32767 ？

---

worker1 与 worker2:

def worker1(proc_num, proc_list, output_queue):
    '''worker function which deadlocks'''  
    for num in proc_list:
        output_queue.put(factorize_naive(num))

def worker2(proc_num, proc_list, output_queue):
    '''worker function that works'''
    workers_stuff = []

    for num in proc_list:
        workers_stuff.append(factorize_naive(num))
    output_queue.put(workers_stuff)

---

SO 上有很多 类似的问题，但我相信这些问题的核心与所有这些问题明显不同。

相关链接:

• https://sopython.com/canon/82/programs-using-multiprocessing-hang-deadlock-and-never-complete/
• python multiprocessing issues
• python multiprocessing - process hangs on join for large queue
• Process.join() and queue don't work with large numbers
• Python 3 Multiprocessing queue deadlock when calling join before the queue is empty
• Script using multiprocessing module does not terminate
• Why does multiprocessing.Process.join() hang?
• When to call .join() on a process?
• What exactly is Python multiprocessing Module's .join() Method Doing?

Answer 1

The docs对此发出警告:

Warning: As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.

This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.

虽然 Queue 看起来是无界的，但在幕后，排队的项目在内存中缓冲以避免进程间管道过载。在刷新这些内存缓冲区之前，进程无法正常结束。您的 worker1() 将比您的 worker2() 更多的项目放在队列中，仅此而已。请注意，在实现诉诸内存缓冲之前可以排队的项目数量没有定义:它可能因操作系统和 Python 版本而异。

正如文档所建议的，避免这种情况的正常方法是.get() 在您尝试之前关闭队列中的所有项目.join() 进程。正如您所发现的，是否有必要这样做取决于每个工作进程已将多少项目放入队列中的未定义方式。