python - joblib 中的并行函数运行除函数之外的整个代码

Question

我正在使用Python中joblib包中的Parallel函数。我只想使用此函数来处理我的函数之一，但不幸的是整个代码是并行运行的(除了其他函数)。

示例:

from joblib import Parallel, delayed
print ('I do not want this to be printed n times')
def do_something(arg):
    some calculations(arg)

Parallel(n_jobs=5)(delayed(do_something)(i) for i in range(0, n))

Answer 1

这是一个常见的错误，因为它错过了文档中的设计方向。许多用户都遇到过同样的经历。

文档非常清楚地说明在__main__ fuse 之前不放置任何代码，但def-s。

如果不这样做，错误确实会喷出来，事情会变得严重破坏，但是，重新阅读文档的明确建议仍然存在，在屏幕上无限泄漏:

[joblib] Attempting to do parallel computing
without protecting your import on a system that does not support forking.

To use parallel-computing in a script, you must protect your main loop
using "if __name__ == '__main__'".

Please see the joblib documentation on Parallel for more information

<小时/>

解决方案:

正确完成第一期，报告 w.r.t。融合的导入保护，事情会变得更好:

C:\Python27.anaconda>python joblib_example.py
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...
I do not want this to be printed n-times...

接下来最后一步就完成了:

from sklearn.externals.joblib  import Parallel, delayed

def do_some_thing( arg ):
    pass
    return True

if  __name__ == '__main__': #################################### A __main__ FUSE:

    pass;                                   n = 6
    print "I do not want this to be printed n-times..."

    Parallel( n_jobs = 5 ) ( delayed( do_some_thing )( i )
                                                   for i in range( 0, n )
                             )

<小时/>

C:\Python27.anaconda>python joblib_example.py
I do not want this to be printed n-times...

C:\Python27.anaconda>