python - 多处理简单功能不起作用，但为什么

Question

我正在尝试多处理系统命令，但无法让它与简单的程序一起工作。不过，函数 runit(cmd) 工作正常......

#!/usr/bin/python3
from subprocess import call, run, PIPE,Popen
from multiprocessing import Pool
import os
pool = Pool()

def runit(cmd):
    proc = Popen(cmd, shell=True,stdout=PIPE, stderr=PIPE, universal_newlines=True)
    return proc.stdout.read()

#print(runit('ls -l'))

it = []
for i in range(1,3):
    it.append('ls -l')

results = pool.map(runit, it)

它输出:

Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
AttributeError: Can't get attribute 'runit' on <module '__main__' from './syscall.py'>
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
AttributeError: Can't get attribute 'runit' on <module '__main__' from './syscall.py'>

然后它以某种方式等待并且什么也不做，当我按 Ctrl+C 几次时它会吐出:

^CProcess ForkPoolWorker-4:
Process ForkPoolWorker-6:
Traceback (most recent call last):
  File "./syscall.py", line 17, in <module>
Process ForkPoolWorker-5:
    results = pool.map(runit, it)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
...
    buf = self._recv(4)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

Answer 1

我不确定，因为我知道的问题与 Windows 相关(并且我无法访问 Linux 盒子来重现)，但为了便于移植，您必须将依赖于多处理的命令包装在 if __name__=="__main__" 或者它与 python 生成进程的方式冲突:该固定示例在 Windows 上运行良好(并且在其他平台上也应该正常工作):

from multiprocessing import Pool
import os

def runit(cmd):
    proc = Popen(cmd, shell=True,stdout=PIPE, stderr=PIPE, universal_newlines=True)
    return proc.stdout.read()

#print(runit('ls -l'))

it = []
for i in range(1,3):
    it.append('ls -l')

if __name__=="__main__":
    # all calls to multiprocessing module are "protected" by this directive
    pool = Pool()

(更仔细地研究错误消息，现在我很确定只需将 pool = Pool() 移动到 runit 的声明之后即可> 也会在 Linux 上修复它，但包装在 __main__ 修复+使其可移植)

也就是说，请注意，您的多处理只是创建一个新进程，因此您最好使用线程池( Threading pool similar to the multiprocessing Pool? ):创建进程的线程，如下所示:

from multiprocessing.pool import ThreadPool  # uses threads, not processes
import os

def runit(cmd):
    proc = Popen(cmd, shell=True,stdout=PIPE, stderr=PIPE, universal_newlines=True)
    return proc.stdout.read()

it = []
for i in range(1,3):
    it.append('ls -l')

if __name__=="__main__":
    pool = ThreadPool()   # ThreadPool instead of Pool
    results = pool.map(runit, it)
    print(results)
        results = pool.map(runit, it)
        print(results)

后一种解决方案更加轻量级，并且不太容易出现问题(多处理是一个需要处理的微妙模块)。您将能够使用对象、共享数据等...而无需 Manager 对象，还有其他优点