python - 并行化令人尴尬的可并行化生成器的简单方法

Question

我有一个生成器(或生成器列表)。我们称他们为 gens

gens 中的每个生成器都是一个复杂的函数，它返回一个复杂过程的下一个值。幸运的是，它们都是相互独立的。

我想为 gens 中的每个元素 gen 调用 gen.__next__()，并在列表中返回结果值。但是，多处理对酸洗生成器不满意。

在 Python 中是否有一种快速、简单的方法来做到这一点？我希望长度为 m 的 gens 映射到我机器上的本地 n 个内核，其中 n 可以大于或小于 m。每个生成器都应在单独的核心上运行。

如果这是可能的，有人可以提供一个最小的例子吗？

Answer 1

你不能 pickle 生成器。了解更多信息 here .

有一篇博文对其进行了更详细的解释。引用其中的一句话:

Let’s ignore that problem for a moment and look what we would need to do to pickle a generator. Since a generator is essentially a souped-up function, we would need to save its bytecode, which is not guarantee to be backward-compatible between Python’s versions, and its frame, which holds the state of the generator such as local variables, closures and the instruction pointer. And this latter is rather cumbersome to accomplish, since it basically requires to make the whole interpreter picklable. So, any support for pickling generators would require a large number of changes to CPython’s core.

Now if an object unsupported by pickle (e.g., a file handle, a socket, a database connection, etc) occurs in the local variables of a generator, then that generator could not be pickled automatically, regardless of any pickle support for generators we might implement. So in that case, you would still need to provide custom getstate and setstate methods. This problem renders any pickling support for generators rather limited.

他还提出了一个解决方案，即使用简单的迭代器。

the best solution to this problem to the rewrite the generators as simple iterators (i.e., one with a __next__ method). Iterators are easy and efficient space-wise to pickle because their state is explicit. You would still need to handle objects representing some external state explicitly however; you cannot get around this.

另提供solution (我没试过)建议这个

1. 将生成器转换为生成器代码为__iter__方法的类

2. 将__getstate__ 和__setstate__ 方法添加到类中，以处理pickling。请记住，您不能 pickle 文件对象。因此 __setstate__ 将不得不根据需要重新打开文件。