python - 池映射未使用所有可用资源的可能原因

Question

我正在运行以下代码

from multiprocessing import Pool


def loop_f(x, num_loops):
    for i in range(num_loops):
        f(x)
    return 

def f(x):
    result = 0 
    for i in range(x):
        result = result*i
    return result

x = 200000
num_times=200
for i in range(8):
    p = Pool(i +1)
    print(i+1)
    %time res=p.map(f, [x]*num_times)

现在，当我运行这段代码时，我发现性能改进在第 4 个进程后停止了

Timing when using  1  processes
CPU times: user 9.08 ms, sys: 13.4 ms, total: 22.5 ms
Wall time: 1.17 s
Timing when using  2  processes
CPU times: user 0 ns, sys: 12.1 ms, total: 12.1 ms
Wall time: 598 ms
Timing when using  3  processes
CPU times: user 5.51 ms, sys: 5.6 ms, total: 11.1 ms
Wall time: 467 ms
Timing when using  4  processes
CPU times: user 9.1 ms, sys: 479 µs, total: 9.58 ms
Wall time: 348 ms
Timing when using  5  processes
CPU times: user 4.15 ms, sys: 4.51 ms, total: 8.66 ms
Wall time: 352 ms
Timing when using  6  processes
CPU times: user 6.85 ms, sys: 2.74 ms, total: 9.59 ms
Wall time: 343 ms
Timing when using  7  processes
CPU times: user 2.79 ms, sys: 7.16 ms, total: 9.95 ms
Wall time: 349 ms
Timing when using  8  processes
CPU times: user 9.06 ms, sys: 427 µs, total: 9.49 ms
Wall time: 362 ms

但是当我检查我的系统时，我应该可以访问至少 8 个处理器内核。

import multiprocessing
import os

print(multiprocessing.cpu_count())
print(len(os.sched_getaffinity(0)))

8
8

那么发生了什么，或者可能发生了什么？如何最大限度地提高系统性能？

Answer 1

您应该只创建一次池。

from multiprocessing import Pool

def f(x):
    j = 0
    for i in range(1000000):
        j += i

    return x*x

if __name__ == '__main__':
    with Pool(8) as p:
        print(p.map(f, range(1000)))

上面的代码让我的八个线程忙了一会儿。