python - Pandas 与 pickle 0.14.1 和 0.15.2 的向后兼容性问题

Question

我们使用 pandas Dataframe 作为我们的时间序列数据的主要数据容器。我们将数据帧打包成二进制 blob 到一个 mongoDB 文档中进行存储，以及有关时间序列 blob 的元数据的键。

我们从 pandas 0.14.1 升级到 0.15.2 时遇到了错误。

创建 pandas Dataframe (0.14.1) 的二进制 blob

import lz4   
import cPickle

bd = lz4.compress(cPickle.dumps(df,cPickle.HIGHEST_PROTOCOL))

错误案例:使用 pandas 0.15.2 从 mongoDB 回读

cPickle.loads(lz4.decompress(bd))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-37-76f7b0b41426> in <module>()
----> 1 cPickle.loads(lz4.decompress(bd))
TypeError: ('_reconstruct: First argument must be a sub-type of ndarray', <built-in function _reconstruct>, (<class 'pandas.core.index.Index'>, (0,), 'b'))

成功案例:使用 pandas 0.14.1 从 mongoDB 回读，没有错误。

这似乎类似于旧堆栈线程 Pandas compiled from source: default pickle behavior changed来自 https://stackoverflow.com/users/644898/jeff 的有用评论

The error message you are seeing `TypeError: _reconstruct: First argument must be a sub-type of ndarray is that the python default unpickler makes sure that the class hierarchy that was pickled is exactly the same what it is recreating. Since Series has changed between versions this is no longer possible with the default unpickler, (this IMHO is a bug in the way pickle works). In any event, pandas will unpickle pre-0.13 pickles that have Series objects."

关于变通方法或解决方案的任何想法？

重现错误:

pandas 0.14.1 环境中的设置:

df = pd.DataFrame(np.random.randn(10,10))
cPickle.dump(df,open("cp0141.p","wb"))
cPickle.load(open('cp0141.p','r')) # no error

在 pandas 0.15.2 env 中创建错误:

cPickle.load(open('cp0141.p','r'))
TypeError: ('_reconstruct: First argument must be a sub-type of ndarray', <built-in function_reconstruct>, (<class 'pandas.core.index.Int64Index'>, (0,), 'b'))

Answer 1

这是明确提到的 Index 类现在不再是 ndarray 的子类，而是一个 pandas 对象，参见 here .

您只需要使用 pd.read_pickle 来读取 pickle 。