python - 错误或功能？无法在 python 脚本中执行两个连续的多处理步骤

Question

我有两个处理大型列表的连续函数。

我一个接一个地打电话，使用 joblib 的Parallel, delayed , 试图分别提高两个函数的处理速度。

但是，我看到了 function_1 的输出也尽快Parallel电话 function_2我不明白为什么。 简而言之，这会导致 function_2没有被调用。

主要代码:

from mycode import function_2
from joblib import Parallel, delayed
import gc

if __name__ == '__main__':  
   list = list_1
   print ">>> First call"
   Parallel(n_jobs = -1)(delayed(function_1) 
                                         (item) for item in list)
   gc.collect()
   do_other_stuff()
   list = list_2
   print ">>> Second call"
   Parallel(n_jobs=-1, backend='threading')(delayed(function_2)
                                         (item) for item in list)

线程函数:

def function_1(): # Gets called first
    print "this comes from function 1"
    pass

def function_2(): # Gets called second
    print "this comes from function 2"
    pass

输出:

>>> First call
this comes from function 1
this comes from function 1
this comes from function 1
this comes from function 1
>>> Second call
this comes from function 1
this comes from function 1
this comes from function 1
this comes from function 1

我的假设是 function_1 的某些部分存储在内存中，调用后保留(可能是由于 joblib 内存映射/共享功能？)。

这就是为什么我 gc.collect()在通话之间。由于这没有帮助，我考虑 reloading modules在通话之间(joblib, Parallel, delayed)，这看起来很难看。

有没有人遇到过类似的行为(在 Windows 上)？

是否有一些修复？

我需要卸载/重新加载吗 joblib或 mycode模块在这里，在 Parallel 之间步骤，如果是，为什么？

Answer 1

简短版:

Q1:有没有人遇到过类似的行为(在 Windows 上)？
A1:没有。

Q2:有什么解决办法吗？
A2:不，引用。 A1.

问题 3:我是否需要在此处卸载/重新加载 joblib 或 mycode 模块......？
A3:不，引用。 A1.

Q4:如果是这样(Q3)，为什么？
A4:不适用，引用。 A3.

---

让我们基于一个通用的 MCVE 公式来插入我们的努力:

稍作修改的实验结构可能如下所示:

#ass;                         import function_2
from sklearn.externals.joblib import Parallel, delayed
#ass;                         import gc
pass;                         import itertools

def function_1( aParam = "" ):                                          # Gets called first
    try:
         print "this comes from a call: function_1( aParam == {0:})".format( aParam )

    except:
         pass # die in silence

    finally:
         return aParam

def function_2( aParam = "" ):                                          # Gets called second
    try:
         print "this comes from a call: function_2( aParam == {0:})".format( aParam )

    except:
         pass # die in silence

    finally:
         return aParam

if __name__ == '__main__':
   print "-------------------------------------------------------------- vvv main.START()"
   #ist = list_1
   aList = [ 11, 12, 13, 14, 15, ]
   print "-------------------------------------------------------------- >>> First call"
   A = Parallel(                                               n_jobs = -1
                 )( delayed( function_1 ) ( item ) for item in aList
                    )
   print "-------------------------------------------------------------- vvv main was ret'd: {0:}".format( repr( A ) )
   #c.collect()
   #o_other_stuff()
   #ist = list_2
   aList = [ 21, 22, 23, 24, 25, ]
   print "-------------------------------------------------------------- >>> Second call"
   B = Parallel(                                               n_jobs  = -1,
                                                               backend = 'threading'
                 )( delayed( function_2 ) ( item ) for item in aList
                    )
   print "-------------------------------------------------------------- vvv main was ret'd: {0:}".format( repr( B ) )

---

结果:

C:\Python27.anaconda>python TEST_SO_Parallel.py
-------------------------------------------------------------- vvv main.START()
-------------------------------------------------------------- >>> First call
this comes from a call: function_1( aParam == 11)
this comes from a call: function_1( aParam == 12)
this comes from a call: function_1( aParam == 13)
this comes from a call: function_1( aParam == 14)
this comes from a call: function_1( aParam == 15)
-------------------------------------------------------------- vvv main was ret'd: [11, 12, 13, 14, 15]
-------------------------------------------------------------- >>> Second call
this comes from a call: function_2( aParam == 21)
this comes from a call: function_2( aParam == 22)
this comes from a call: function_2( aParam == 23)
 this comes from a call: function_2( aParam == 25)this comes from a call: function_2( aParam == 24)

-------------------------------------------------------------- vvv main was ret'd: [21, 22, 23, 24, 25]

---

评估:

据观察，[win] py2.7 已处理代码，没有任何上述报告的障碍。

根据规范，joblib 记录的处理是正确的。

上面报告的行为没有被复制，可以映射到任何形式的 joblib - 参与因果链。