Python:多处理无法完成作业

Question

我正在使用带有 multiprocessing::Pool 的 python 2.7 并行运行作业

我已经简化了下面的例子，但这里是它的主要要点。

它将使用 apply_async() 为我的字典中的每个人创建一个文件功能。但是，当我检查文件是否正确创建时，我注意到有时未创建文件。

现在我想我在使用 multiprocessing::Pool 时做错了什么

有什么建议吗？

import os
from multiprocessing import Pool

def outputFile(person):
    ofh=open(person+'.txt','w')
    ofh.write('test\n')
    ofh.close()

pool = Pool(processes=4)
for person in person_dict:
    pool.apply_async(outputFile,args(person))
pool.close()
pool.join()
for person in person_dict:
    print os.path.isfile(person+'.txt')

True
True
False
True

Answer 1

如果不捕获子流程中的异常并自己打印它们，您将看不到它们。以下程序不产生任何输出:

import os
from multiprocessing import Pool

def outputFile(person):
    raise Exception("An exception")

pool = Pool(processes=4)
for person in range(100):
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

您需要捕获所有异常并手动打印回溯:

import os
from multiprocessing import Pool, Lock
import traceback

print_lock = Lock()

def outputFile(person):
    try:
        raise Exception("An exception")
    except:
        with print_lock:
            print "%s: An exception occurred" % person
            print traceback.format_exc()

pool = Pool(processes=4)
for person in range(100):
    args = (person, print_lock)
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

输出

0: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

1: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

...

99: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

注: print_lock用于防止输出被交错。