python - 并行执行函数列表

Question

因此，使用多进程模块可以轻松地使用不同的参数并行运行函数，如下所示:

from multiprocessing import Pool

def f(x):
    return x**2


p = Pool(2)
print(p.map(f, [1, 2]))

但我有兴趣在同一参数上执行函数列表。假设我有以下两个函数:

def f(x):
    return x**2


def g(x):
    return x**3 + 2

如何针对同一参数(例如 x=1)并行执行它们？

Answer 1

您可以使用Pool.apply_async()来实现这一点。您以 (function, argument_tuple) 的形式捆绑任务，并将每个任务提供给 apply_async()。

from multiprocessing import Pool
from itertools import repeat


def f(x):
    for _ in range(int(50e6)): # dummy computation
        pass
    return x ** 2


def g(x):
    for _ in range(int(50e6)): # dummy computation
        pass
    return x ** 3


def parallelize(n_workers, functions, arguments):
    # if you need this multiple times, instantiate the pool outside and
    # pass it in as dependency to spare recreation all over again
    with Pool(n_workers) as pool:
        tasks = zip(functions, repeat(arguments))
        futures = [pool.apply_async(*t) for t in tasks]
        results = [fut.get() for fut in futures]
    return results


if __name__ == '__main__':

    N_WORKERS = 2

    functions = f, g
    results = parallelize(N_WORKERS, functions, arguments=(10,))
    print(results)

示例输出:

[100, 1000]

Process finished with exit code 0