"Resource temporarily unavailable"后的Python多处理池恢复

Question

如果我在 Python 解释器中创建了一个包含大量进程的池，它显然会出错，但是在这样做之前似乎并没有清理 fork 进程，因此留下了环境脏，系统的其余部分无法 fork 进程。

>>> from multiprocessing import Pool
>>> p = Pool(1000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 159, in __init__
    self._repopulate_pool()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool
    w.start()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()
OSError: [Errno 35] Resource temporarily unavailable

>>> p = Pool(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 159, in __init__
    self._repopulate_pool()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool
    w.start()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()
OSError: [Errno 35] Resource temporarily unavailable

有什么方法可以避免/补救这个问题，或者它是否被认为是一个错误？

Answer 1

Is there some way to avoid/remedy this,

不要那样做。

or is it considered a bug?

是的，如果初始化器失败，所有分配的资源都应该被取消分配。您应该检查您正在使用的 2.7 的特定版本，看看是否有任何多处理特定的库错误在以后的版本中修复(2.7.6 发行说明:http://hg.python.org/cpython/raw-file/99d03261c1ba/Misc/NEWS)。

根据堆栈跟踪中的路径，我假设您的平台是 OSX。这是关于 fork 时错误号 35(在 OSX 中似乎是 EAGAIN)的帖子 - I can't run more than 100 processes

无论您想要完成什么，您似乎都需要在应用程序级别对资源使用进行限制。这意味着您可能需要重新考虑您的解决方案。使用您目前的解决方案并修复错误，您仍然可能会在其他情况下看到系统范围内的资源限制。