python-2.7 - Scons AttributeError : 'builtin_function_or_method' object has no attribute 'dispatch'

Question

我有一个实例化对象的 sconstruct 脚本。这个对象在内部调用一个方法而不是运行多处理模块。下图示例

该对象在调用函数之前解压缩文件并将输入传递给多处理模块。

def run_scons(self,inpfile,outfile):

        # Unpickle input parameter
        fid=open(inpfile,'rb')
        input_data=pkls.load(fid)
        my_results=[]
        #run solver in loop
        for my_data in input_data:
            work_ers=len(my_data)
            pool = Pool(processes=work_ers)
            a_result=pool.map_async(my_solver, my_data)
            pool.close()
            pool.join()
            my_results.append(a_result.get())
        fid.close()

        fid_out=open(outfile,'wb+')
        pkls.dump(rot_full_results,fid_out)

通过 scons 执行相同的功能时出现以下错误。

pool = Pool(processes=work_ers)
  File "C:\Python27\lib\multiprocessing\__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "C:\Python27\lib\multiprocessing\pool.py", line 138, in __init__
    self._setup_queues()
  File "C:\Python27\lib\multiprocessing\pool.py", line 232, in _setup_queues
    from .queues import SimpleQueue
  File "C:\Python27\lib\multiprocessing\queues.py", line 48, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "C:\Python27\lib\multiprocessing\synchronize.py", line 48, in <module>
    from multiprocessing.forking import assert_spawning, Popen
  File "C:\Python27\lib\multiprocessing\forking.py", line 60, in <module>
    class ForkingPickler(Pickler):
  File "C:\Python27\lib\multiprocessing\forking.py", line 61, in ForkingPickler
    dispatch = Pickler.dispatch.copy()
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'
scons: building terminated because of errors.

在阅读了这个错误之后，我发现 SCONS 有一个技巧，它将 pickle 模块重命名为 cPickle，而多处理模块正在寻找 cPickle 并且一切都失败了。有没有办法解决？

Answer 1

我正在使用多处理并行处理一些“作业”文件。我在我的 SConstruct 文件中做这样的事情:

if GetOption("run_jobs"):

    my_jobs = # some code that produces a list of job objects

    ecode = build_support.mp_run_jobs(my_jobs)

    Exit(ecode)

但后来我遇到了上面的异常:

AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'

在 Linux 上使用 'SCONS_HORRIBLE_REGRESSION_TEST_HACK' 有效:

$ SCONS_HORRIBLE_REGRESSION_TEST_HACK=1 scons --run-jobs

但我不想在 Linux、Windows 和 Mac 等设备上导出该符号。

我确实找到了一种解决方法，删除泡菜模块并像这样重新导入它们:

if GetOption("run_jobs"):

    # Workaround SCons.compat module renaming

    import imp

    del sys.modules['pickle']
    del sys.modules['cPickle']

    sys.modules['pickle'] = imp.load_module('pickle', *imp.find_module('pickle'))
    sys.modules['cPickle'] = imp.load_module('cPickle', *imp.find_module('cPickle'))

    import pickle
    import cPickle

    print "(pickle == cPickle) = ", (pickle == cPickle)

    my_jobs = # some code that produces a list of job objects

    ecode = build_support.mp_run_jobs(my_jobs)

    Exit(ecode)

现在我的作业多处理按预期工作!