gpt4 book ai didi

python - 创建对象的副本而不是在新的多处理进程内部重新初始化

转载 作者:太空宇宙 更新时间:2023-11-04 01:35:41 24 4
gpt4 key购买 nike

这段代码显示了我正在尝试做的事情的结构。

import multiprocessing
from foo import really_expensive_to_compute_object

## Create a really complicated object that is *hard* to initialise.
T = really_expensive_to_compute_object(10)

def f(x):
return T.cheap_calculation(x)

P = multiprocessing.Pool(processes=64)
results = P.map(f, range(1000000))

print results

问题是每个进程开始时都会花费大量时间重新计算 T,而不是使用计算过一次的原始 T。有没有办法防止这种情况? T 有一个快速(深)复制方法,那么我可以让 Python 使用它而不是重新计算吗?

最佳答案

multiprocessing文档 suggests

Explicitly pass resources to child processes

所以你的代码可以重写成这样:

import multiprocessing
import time
import functools

class really_expensive_to_compute_object(object):
def __init__(self, arg):
print 'expensive creation'
time.sleep(3)

def cheap_calculation(self, x):
return x * 2

def f(T, x):
return T.cheap_calculation(x)

if __name__ == '__main__':
## Create a really complicated object that is *hard* to initialise.
T = really_expensive_to_compute_object(10)
## helper, to pass expensive object to function
f_helper = functools.partial(f, T)
# i've reduced count for tests
P = multiprocessing.Pool(processes=4)
results = P.map(f_helper, range(100))

print results

关于python - 创建对象的副本而不是在新的多处理进程内部重新初始化,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/10055276/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com