python - 在 python 中压缩大数据的问题

Question

我有一个 Python 脚本来压缩大字符串:

import zlib

def processFiles():
  ...
  s = """Large string more than 2Gb"""
  data = zlib.compress(s)
  ...

当我运行这个脚本时，我得到一个错误:

ERROR: Traceback (most recent call last):#012  File "./../commands/sce.py", line 438, in processFiles#012    data = zlib.compress(s)#012OverflowError: size does not fit in an int

一些信息:

zlib.版本 = '1.0'

zlib.ZLIB_VERSION = '1.2.7'

# python -V
Python 2.7.3

# uname -a
Linux app2 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux

# free
             total       used       free     shared    buffers     cached
Mem:      65997404    8096588   57900816          0     184260    7212252
-/+ buffers/cache:     700076   65297328
Swap:     35562236          0   35562236

# ldconfig -p | grep python
libpython2.7.so.1.0 (libc6,x86-64) => /usr/lib/libpython2.7.so.1.0
libpython2.7.so (libc6,x86-64) => /usr/lib/libpython2.7.so

如何在 Python 中压缩大数据(超过 2Gb)？

Answer 1

我的压缩大数据的函数:

def compressData(self, s):
    compressed = ''
    begin = 0
    blockSize = 1073741824 # 1Gb
    compressor = zlib.compressobj()
    while begin < len(s):
      compressed = compressed + compressor.compress(s[begin:begin + blockSize])
      begin = begin + blockSize
    compressed = compressed + compressor.flush()
    return compressed