Python 多重处理在遵循文档代码时返回 AttributeError

Question

这个问题已经有答案了:

python multiprocessing in Jupyter on Windows: AttributeError: Can't get attribute "abc" (4 个回答)

已关闭 4 年前。

我决定尝试进入多处理器模块来帮助加速我的程序。为了弄清楚这一点，我尝试使用有关多处理的官方 python 文档中的一些代码示例。

第一次尝试:Introduction

>>> from multiprocessing import Pool
>>>
>>> def f(x):
...     return x*x
...
>>> if __name__ == '__main__':
...     with Pool(5) as p:
...         print(p.map(f, [1, 2, 3]))
...
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Program Files\Python36\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Program Files\Python36\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Process SpawnPoolWorker-4:
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\Program Files\Python36\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

这里我假设池函数被破坏了；也许最新版本中某个地方有错别字。所以我尝试了一些更具体的代码。

第二次尝试:Process class code block 2

>>> from multiprocessing import Process
>>> import os
>>>
>>> def info(title):
...     print(title)
...     print('module name:', __name__)
...     print('parent process:', os.getppid())
...     print('process id:', os.getpid())
...
>>> def f(name):
...     info('function f')
...     print('hello', name)
...
>>> if __name__ == '__main__':
...     info('main line')
...     p = Process(target=f, args=('bob',))
...     p.start()
...     p.join()
...
main line
module name: __main__
parent process: 43824
process id: 54888
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

此时我知道潜在的错误与多处理的 Process 函数有关。但是，我认为扩展代码可能会破坏某些内容，因此我尝试了简单的代码。

第三次尝试 Process class code block 1

>>> from multiprocessing import Process
>>>
>>> def f(name):
...     print('hello', name)
...
>>> if __name__ == '__main__':
...     p = Process(target=f, args=('bob',))
...     p.start()
...     p.join()
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

此时我已经绝望了。我认为这个争论可能会扰乱 Process 类。

最终尝试:自行生成代码

>>> from multiprocessing import Process
>>>
>>> def f():
...     print('hello')
...
>>> if __name__ == '__main__':
...     p = Process(target=f)
...     p.start()
...     p.join()
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

现在我完全困惑了，因为我不知道为什么会发生错误。有人可以帮我弄清楚为什么我每次都会收到此错误吗？

Answer 1

您正处于交互模式。这基本上不适用于多处理，因为工作人员必须导入 __main__ 并获取与主进程的 __main__ 类似的东西。这是多处理 API 令人极其困惑的众多方式之一。

将代码放入脚本中并运行该脚本。