python - 加载 Pickle 文件时出现问题，内存错误

Question

我想加载一个pickle，它在Python 2.7中工作，但在Python 3.6中不起作用加载时出现内存错误。我尝试将pickle重新格式化为python 3格式，然后尝试加载它，但问题仍然存在，

使用 Windows 10 笔记本电脑、8GB RAM 时，pickle 文件的大小为 1.1GB。该 pickle 有 14804726 行和 10 列。

关于如何进一步解决这个问题的任何线索？

import pandas as pd

def readpickle(picklefile):
    rawdata = pd.read_pickle(picklefile)    
    return rawdata

picklefile=rawdata_py3.pkl' 
readpickle(picklefile)

错误:

Traceback (most recent call last):

  File "<ipython-input-3-3fea1d423973>", line 1, in <module>
    runfile('D:/PROJECTS/FR24/Scripts/pickletest.py', wdir='D:/PROJECTS/FR24/Scripts')

  File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 11, in <module>
    readpickle(picklefile)

  File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 7, in readpickle
    rawdata = pd.read_pickle(picklefile) #to load pickle

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 180, in read_pickle
    return try_read(path, encoding='latin1')

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in try_read
    lambda f: pc.load(f, encoding=encoding, compat=True))

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 149, in read_wrapper
    return func(f)

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in <lambda>
    lambda f: pc.load(f, encoding=encoding, compat=True))

  File "c:\python36\lib\site-packages\pandas\compat\pickle_compat.py", line 212, in load
    return up.load()

  File "c:\python36\lib\pickle.py", line 1050, in load
    dispatch[key[0]](self)

  File "c:\python36\lib\pickle.py", line 1077, in load_frame
    self._unframer.load_frame(frame_size)

  File "c:\python36\lib\pickle.py", line 257, in load_frame
    self.current_frame = io.BytesIO(self.file_read(frame_size))

MemoryError

Traceback (most recent call last):

  File "<ipython-input-3-3fea1d423973>", line 1, in <module>
    runfile('D:/PROJECTS/FR24/Scripts/pickletest.py', wdir='D:/PROJECTS/FR24/Scripts')

  File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "c:\python36\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 11, in <module>
    readpickle(picklefile)

  File "D:/PROJECTS/FR24/Scripts/pickletest.py", line 7, in readpickle
    rawdata = pd.read_pickle(picklefile) #to load pickle

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 180, in read_pickle
    return try_read(path, encoding='latin1')

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in try_read
    lambda f: pc.load(f, encoding=encoding, compat=True))

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 149, in read_wrapper
    return func(f)

  File "c:\python36\lib\site-packages\pandas\io\pickle.py", line 175, in <lambda>
    lambda f: pc.load(f, encoding=encoding, compat=True))

  File "c:\python36\lib\site-packages\pandas\compat\pickle_compat.py", line 212, in load
    return up.load()

  File "c:\python36\lib\pickle.py", line 1050, in load
    dispatch[key[0]](self)

  File "c:\python36\lib\pickle.py", line 1077, in load_frame
    self._unframer.load_frame(frame_size)

  File "c:\python36\lib\pickle.py", line 257, in load_frame
    self.current_frame = io.BytesIO(self.file_read(frame_size))

MemoryError 

Answer 1

所以最后我弄清楚了问题是什么，它在 python 2.7 上工作，但在 python 3.6 上不起作用，在比较这些时，我注意到 python 3.6 是 32 位，而 python 2.7 版本是 64 位。因此，在卸载并重新安装 64 位 python 3.6 版本后，pickle 加载得很好。