python - 如何分块加载 Pickle 文件？

Question

是否有任何选项可以分块加载 pickle 文件？

我知道我们可以将数据保存在 CSV 中并分块加载。
但是除了 CSV 之外，是否有任何选项可以分块加载 pickle 文件或任何 Python native 文件？

Answer 1

基于 the documentation for Python pickle ，目前不支持分块。
但是，可以将数据拆分为块，然后分块读取。例如，假设原始结构是

import pickle

filename = "myfile.pkl"
str_to_save = "myname"

with open(filename,'wb') as file_handle:
    pickle.dump(str_to_save, file_handle)
    
with open(filename,'rb') as file_handle:
    result = pickle.load(file_handle)

print(result)

这可以分成两个单独的 pickle 文件:

import pickle

filename_1 = "myfile_1.pkl"
filename_2 = "myfile_2.pkl"
str_to_save = "myname"

with open(filename_1,'wb') as file_handle:
    pickle.dump(str_to_save[0:4], file_handle)
with open(filename_2,'wb') as file_handle:
    pickle.dump(str_to_save[4:], file_handle)
    
with open(filename_1,'rb') as file_handle:
    result = pickle.load(file_handle)

print(result)

根据 AKX 的评论，将多个数据写入单个文件也有效:

import pickle

filename = "myfile.pkl"
str_to_save = "myname"

with open(filename,'wb') as file_handle:
    pickle.dump(str_to_save[0:4], file_handle)
    pickle.dump(str_to_save[4:], file_handle)
    
with open(filename,'rb') as file_handle:
    result = pickle.load(file_handle)
    print(result)
    result = pickle.load(file_handle)
    print(result)