python - 使用 lambda 序列化 cython 类时出现 pickle 错误

Question

我使用 pickle 和 dill 来跟踪 lambda 函数并且工作正常:

import dill
import pickle

f = lambda x,y: x+y
s = pickle.dumps(f)

甚至在类里面使用时，例如:

文件

foo.py

class Foo(object):
    def __init__(self):
        self.f = lambda x, y: x+y 

文件

test.py

import dill
import pickle
from foo import Foo

f = Foo()
s = pickle.dumps(f)  # or  s = dill.dumps(f)

但是当以 格式构建相同的文件时.pyx (foo.pyx) 使用 cython，不能用 dill、pickle 或 cpickle 序列化，得到这个错误:

Traceback (most recent call last): File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_cod exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in a = pickle.dumps(c) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 1380, in dumps Pickler(file, protocol).dump(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 224, in dump self.save(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 425, in save_reduce save(state) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/dill/_dill.py", line 912, in save_module_dict StockPickler.save_dict(pickler, obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 669, in _batch_setitems save(v) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 317, in save self.save_global(obj, rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 754, in save_global (obj, module, name)) PicklingError: Can't pickle . at 0x7f9ab1ff07d0>: it's not found as foo.lambda

用于构建 cython 的 setup.py 文件

setup.py

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules=cythonize("foo.pyx"))

然后在终端运行:

python setup.py build_ext --inplace

有办法吗？

Answer 1

我是dill作者。扩展@DavidW 在评论中所说的内容——我相信(目前)没有已知的序列化程序可以pickle cython lambdas 或绝大多数cython 代码。实际上，除非 C 扩展代码的作者专门构建序列化指令(如 numpy 和 pandas )，否则 Python 序列化程序要能够使用 C 扩展 pickle 对象要困难得多。在这种情况下...而不是 lambda，您可以使用 __call__ 构建一个类方法，所以它就像一个函数......然后添加一个或多个pickle方法( __reduce__ ， __getstate__ ， __setstate__ 或类似的东西)......然后你应该能够pickle实例你的类(class)。这是一项工作，但由于此路径已用于 pickle 用 C++ 编写的类——我相信您应该能够让它为 cython 构建的类工作。