gpt4 book ai didi

python - 使用并发 future 将多线程 Python 转换为多进程 Python

转载 作者:太空宇宙 更新时间:2023-11-03 17:06:06 26 4
gpt4 key购买 nike

我有以下工作代码(Python 3.5),它使用并发 future 以线程方式解析文件,然后在结果返回时(以任何顺序)对结果进行一些后处理。

from concurrent import futures
with futures.ThreadPoolExecutor(max_workers=4) as executor:
# A dictionary which will contain a list the future info in the key, and the filename in the value
jobs = {}

# Loop through the files, and run the parse function for each file, sending the file-name to it, along with the kwargs of parser_variables.
# The results of the functions can come back in any order.
for this_file in files_list:
job = executor.submit(parse_log_file.parse, this_file, **parser_variables)
jobs[job] = this_file

# Get the completed jobs whenever they are done
for job in futures.as_completed(jobs):
debug.checkpointer("Multi-threaded Parsing File finishing")

# Send the result of the file the job is based on (jobs[job]) and the job (job.result)
result_content = job.result()
this_file = jobs[job]

我想将其转换为使用进程而不是线程,因为线程不提供任何加速。理论上我只需要将 ThreadPoolExecutor 更改为 ProcessPoolExecutor 即可。问题是,如果我这样做,我会得到这个异常:

Process Process-2:
Traceback (most recent call last):
File "C:\Python35\lib\multiprocessing\process.py", line 254, in _bootstrap
self.run()
File "C:\Python35\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Python35\lib\concurrent\futures\process.py", line 169, in _process_worker
call_item = call_queue.get(block=True)
File "C:\Python35\lib\multiprocessing\queues.py", line 113, in get
return ForkingPickler.loads(res)
TypeError: Required argument 'fileno' (pos 1) not found
Traceback (most recent call last):
File "c:/myscript/main.py", line 89, in <module>
main()
File "c:/myscript/main.py", line 59, in main
system_counters = process_system(system, filename)
File "c:\myscript\per_system.py", line 208, in process_system
system_counters = process_filelist(**file_handling_variables)
File "c:\myscript\per_logfile.py", line 31, in process_filelist
results_list = job.result()
File "C:\Python35\lib\concurrent\futures\_base.py", line 398, in result
return self.__get_result()
File "C:\Python35\lib\concurrent\futures\_base.py", line 357, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

我认为这可能与 pickle 有关,但谷歌搜索错误没有发现任何东西。

如何将上述内容转换为使用多个进程?

最佳答案

事实证明,这是因为我在 parser_variables 中传递的内容之一是一个类(来自第三方模块的读取器)。如果我删除该类,上面的方法就可以正常工作。无论出于何种原因,pickle 似乎无法处理这个特定的对象。

关于python - 使用并发 future 将多线程 Python 转换为多进程 Python,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34577083/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com