python-2.7 -> 90％的时间花费在 'acquire'对象的 'thread.lock'方法上

Question

为了确定使用大部分计算时间的步骤，我运行了cProfile并得到以下结果:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.014    0.014  216.025  216.025 func_poolasync.py:2(<module>)
    11241  196.589    0.017  196.589    0.017 {method 'acquire' of 'thread.lock' objects}
      982    0.010    0.000  196.532    0.200 threading.py:309(wait)
     1000    0.002    0.000  196.498    0.196 pool.py:565(get)
     1000    0.005    0.000  196.496    0.196 pool.py:557(wait)
515856/3987    0.350    0.000   13.434    0.003 artist.py:230(stale)

大部分时间显然都花在了这个步骤上，这似乎是 method 'acquire' of 'thread.lock' objects。我没有使用线程；而是我将 pool.apply_async与几个处理器一起使用，所以我很困惑为什么 thread.lock是问题？

我希望能弄清楚为什么这是瓶颈？以及如何降低这次？

代码如下所示:

path='/usr/home/work'
filename='filename'

with open(path+filename+'/'+'result.pickle', 'rb') as f:
     pdata = pickle.load(f)

if __name__ == '__main__':
    pool = Pool()    
    results=[]
    data=list(range(1000))
    print('START')
    start_time = int(round(time.time()))
    result_objects = [pool.apply_async(func, args=(nomb,pdata[0],pdata[1],pdata[2])) for nomb in data]

    results = [r.get() for r in result_objects]

    pool.close()
    pool.join()
    print('END', int(round(time.time()))-start_time)

修订版:

通过从 pool.apply_async切换到 pool.map，我可以将执行时间减少约3倍。

输出:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.113    0.113   70.824   70.824 func.py:2(<module>)
     4329   28.048    0.006   28.048    0.006 {method 'acquire' of 'thread.lock' objects}
        4    0.000    0.000   28.045    7.011 threading.py:309(wait)
        1    0.000    0.000   28.044   28.044 pool.py:248(map)
        1    0.000    0.000   28.044   28.044 pool.py:565(get)
        1    0.000    0.000   28.044   28.044 pool.py:557(wait)

修改代码:

if __name__ == '__main__':
    pool = Pool()    
    data=list(range(1000))

    print('START')
    start_time = int(round(time.time()))
    funct = partial(func,pdata[0],pdata[1],pdata[2])
    results = pool.map(funct,data)
    print('END', int(round(time.time()))-start_time)

但是，已经发现 的某些迭代会导致无意义的结果。我不知道为什么会这样，但是可以看到
速率确定步骤仍然是“thread.lock”对象的“方法”获取。

Answer 1

The profiler is only telling you what the main process is doing, while its child processes are doing all the work.

资料来源: https://stackoverflow.com/a/22530665/1064565