python - 在没有继承的情况下在多个进程之间共享numpy数组

Question

我想在多个进程之间共享 numpy 数组。有可行的解决方案here .但是他们都是通过继承将数组传递给子进程，这对我不起作用，因为我必须事先启动几个工作进程，我不知道以后要处理多少数组。有没有办法在进程启动后创建这样的数组，并通过队列将这些数组传递给进程？

顺便说一句，由于某些原因我无法使用 multiprocessing.Manager。

Answer 1

你应该使用 shared memory ，这完全解决了您的用例。您可以保持内存读/写速度，所有进程都可以在共享内存中读取和写入数组，而不会产生任何序列化或传输成本。

下面是来自官方 python 文档的示例:

>>> # In the first Python interactive shell
>>> import numpy as np
>>> a = np.array([1, 1, 2, 3, 5, 8])  # Start with an existing NumPy array
>>> from multiprocessing import shared_memory
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
>>> # Now create a NumPy array backed by shared memory
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
>>> b[:] = a[:]  # Copy the original data into shared memory
>>> b
array([1, 1, 2, 3, 5, 8])
>>> type(b)
<class 'numpy.ndarray'>
>>> type(a)
<class 'numpy.ndarray'>
>>> shm.name  # We did not specify a name so one was chosen for us
'psm_21467_46075'

>>> # In either the same shell or a new Python shell on the same machine
>>> import numpy as np
>>> from multiprocessing import shared_memory
>>> # Attach to the existing shared memory block
>>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075')
>>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example
>>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf)
>>> c
array([1, 1, 2, 3, 5, 8])
>>> c[-1] = 888
>>> c
array([  1,   1,   2,   3,   5, 888])

>>> # Back in the first Python interactive shell, b reflects this change
>>> b
array([  1,   1,   2,   3,   5, 888])

>>> # Clean up from within the second Python shell
>>> del c  # Unnecessary; merely emphasizing the array is no longer used
>>> existing_shm.close()

>>> # Clean up from within the first Python shell
>>> del b  # Unnecessary; merely emphasizing the array is no longer used
>>> shm.close()
>>> shm.unlink()  # Free and release the shared memory block at the very end

对于像您这样的真实用例，您需要使用 Pipe 或任何其他多处理通信机制传递名称 shm.name。请注意，只有这个小字符串需要在进程之间交换；实际数据保留在共享内存空间中。