Python ProcessPoolExecutor 与 QueueHandler

Question

我想在多进程环境中记录到单个文件。我可以得到 sample code从 python 日志记录手册开始工作。但是当我替换为 ProcessPoolExecutor 时，它不起作用。

#    workers = []
#    for i in range(4):
#        worker = multiprocessing.Process(target=worker_process,
#                                         args=(queue, worker_configurer))
#        workers.append(worker)
#        worker.start()
#    for w in workers:
#        w.join()

import concurrent.futures
with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
    for i in range(4):
        executor.submit(worker_process, queue, worker_configurer)
        print(f'submit worker {i}')

Answer 1

您需要稍微更改配方中创建队列的方式。而不是

queue = multiprocessing.Queue(-1)

做

queue = multiprocessing.Manager().Queue(-1)

然后它就应该可以工作了。在我的系统上:

Worker started: ForkProcess-3
Worker started: ForkProcess-4
Worker started: ForkProcess-5
Worker started: ForkProcess-6
Worker finished: ForkProcess-5
Worker finished: ForkProcess-4
Worker finished: ForkProcess-6
Worker finished: ForkProcess-3

更新:我更新了文档中的食谱配方以描述 concurrent.futures 的用法。重建文档后，更新应该会显示。