Python:fromfile 数组太大和/或 gzip 不会写入

Question

我的目标是从更大的未压缩图像(30000, 65536)中提取大的子图像，而不将整个图像读入内存，然后以压缩格式保存子图像。目前，我只关心确定压缩效果如何作为图像复杂性的指标；我不需要以可见格式保存图像，但我很乐意这样做。这是我的第一个 python 脚本，我陷入了某些函数调用的入口限制。

根据两次替代尝试，我收到两个相关错误(删除了无聊的行):

版本 1:

fd = open(fname,'rb')
h5file=h5py.File(h5fname, "w")
data = h5file.create_dataset("data", (Height, Width), dtype='i', maxshape=(None, None)) # try i8?
data = fromfile(file=fd, dtype='h', count=Height*Width)  #FAIL
fd.close()
h5file.close()
outfilez = gzip.open(outfilename,'wb')
outfilez.write(data)
outfilez.close()

错误:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:...\sitecustomize.py", line 523, in runfile
    execfile(filename, namespace)
  File "C:...\Script_v3.py", line 183, in <module>
    data = fromfile(file=fd, dtype=datatype, count=BandHeight_tracks*Width)
ValueError: array is too big.

版本 2(for 循环以减少 fromfile 使用):

fd = open(fname,'rb')
h5file=h5py.File(h5fname, "w")
data = h5file.create_dataset("data", (Height, Width), dtype='i', maxshape=(None, None)) # try i8?
for i in range(0, Height-1):
    data[i:] = fromfile(file=fd, dtype='h', count=Width)
fd.close()
h5file.close()
outfilez = gzip.open(outfilename,'wb')
outfilez.write(data)
outfilez.close()

错误(我在其他版本中没有得到这个):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:...\sitecustomize.py", line 523, in runfile
    execfile(filename, namespace)
  File "C:...\Script_v4.py", line 195, in <module>
    outfilez.write(data)
  File "C:...\gzip.py", line 235, in write
    self.crc = zlib.crc32(data, self.crc) & 0xffffffffL
TypeError: must be string or read-only buffer, not Dataset

我正在具有 16GB RAM 的 64 位 Win7 计算机上使用 spy 程序运行代码。图片最大为 4GB。

Answer 1

您应该查看 NumPy 中的数组是否存在适合您的需求:

NumPy is a general-purpose array-processing package designed to efficiently manipulate large multi-dimensional arrays of arbitrary records without sacrificing too much speed for small multi-dimensional arrays. NumPy is built on the Numeric code base and adds features introduced by numarray as well as an extended C-API and the ability to create arrays of arbitrary type which also makes NumPy suitable for interfacing with general-purpose data-base applications.