python - 使用多处理模块的脚本不会终止

Question

下面的代码，不打印"here"。问题是什么？我在我的两台机器(Windows 7、Ubuntu 12.10)和 http://www.compileonline.com/execute_python_online.php它不会在所有情况下都打印 "here"。

from multiprocessing import Queue, Process


def runLang(que):
    print "start"
    myDict=dict()
    for i in xrange(10000):
        myDict[i]=i
    que.put(myDict)
    print "finish"


def run(fileToAnalyze):
    que=Queue()
    processList=[]
    dicList=[]
    langs= ["chi","eng"]
    for lang in langs:
        p=Process(target=runLang,args=(que,))
        processList.append(p)
        p.start()

    for p1 in processList:
        p1.join()

    print "here"

    for _ in xrange(len(langs)):
        item=que.get()
        print item
        dicList.append(item)

if __name__=="__main__":
    processList = []
    for fileToAnalyse in ["abc.txt","def.txt"]:
        p=Process(target=run,args=(fileToAnalyse,))
        processList.append(p)
        p.start()
    for p1 in processList:
        p1.join()

Answer 1

这是因为当您将大量项目放入multiprocessing.Queue时，它们最终会在内存中缓冲，一旦底层Pipe已满。在开始从 Queue 的另一端读取数据之前，缓冲区不会被刷新，这将允许 Pipe 接受更多数据。在所有 Queue 实例的缓冲区完全刷新到其底层 Pipe 之前，Process 无法终止。这意味着如果您尝试加入一个进程而没有另一个进程/线程在其Queue上调用get，您可能会死锁。这是 mentioned in the docs :

Warning

As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.

This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.

Note that a queue created using a manager does not have this issue.

您可以通过在清空父级中的 Queue 之前不调用 join 来解决此问题:

for _ in xrange(len(langs)):
    item = que.get()
    print(item)
    dicList.append(item)

# join after emptying the queue.
for p in processList:
    p.join()

print("here")