python - 将 Pandas 数据框保存到 excel

Question

我正在生成一个大型数据框(以 CSV 格式保存时为 1.5 GB)，需要将其存储为 Excel 文件的工作表以及保存在单独工作表中的第二个(小得多的)数据框。

print('Reading temporaty files for variable {}:'.format(Var))
print(' Reading stations')
s=pd.read_csv(StatFile,sep=':',dtype={'ID': 'str'},encoding='utf-8')
print(' Reading data')
d=pd.read_csv(DataFile,sep=':',dtype='str',encoding='utf-8').transpose()
d.columns = d.iloc[0]
d=d[1:].astype('float')
d.reindex_axis(sorted(d.columns), axis=1)
print('Writing out Excel file for variable {}'.format(Var))
writer = pd.ExcelWriter(Path + Var + '.xlsx', engine='xlsxwriter')
d.to_excel(writer, sheet_name='Data')
OutStatCol=['ID','Name','Longitude','Latitude','GRS','OriginalVariable','VariableUnits','URL','JsonNode']
s.to_excel(writer, columns=OutStatCol, index=False, sheet_name='Stations')
writer.save()

我的代码适用于较小的数据框，但对于较大的数据框，我会收到以下错误:

Traceback (most recent call last):
  File "./Test2.py", line 29, in <module>
    writer.save()
  File "/home/user/miniconda2/lib/python2.7/site-packages/pandas/io/excel.py", line 1413, in save
    return self.book.close()
  File "/home/user/miniconda2/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 297, in close
    self._store_workbook()
  File "/home/user/miniconda2/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 624, in _store_workbook
    xlsx_file.write(os_filename, xml_filename)
  File "/home/user/miniconda2/lib/python2.7/zipfile.py", line 1148, in write
    self._writecheck(zinfo)
  File "/home/user/miniconda2/lib/python2.7/zipfile.py", line 1114, in _writecheck
    " would require ZIP64 extensions")
zipfile.LargeZipFile: Filesize would require ZIP64 extensions

有什么方法可以在 ExcelWriter 声明或 to_excel() 方法中指定类似 allowZip64=True 的内容？

谢谢!

Answer 1

这需要一些源代码挖掘，但是......

print('Reading temporaty files for variable {}:'.format(Var))
print(' Reading stations')
s=pd.read_csv(StatFile,sep=':',dtype={'ID': 'str'},encoding='utf-8')
print(' Reading data')
d=pd.read_csv(DataFile,sep=':',dtype='str',encoding='utf-8').transpose()
d.columns = d.iloc[0]
d=d[1:].astype('float')
d.reindex_axis(sorted(d.columns), axis=1)
print('Writing out Excel file for variable {}'.format(Var))
writer = pd.ExcelWriter(Path + Var + '.xlsx', engine='xlsxwriter')

#THIS
writer.book.use_zip64()

d.to_excel(writer, sheet_name='Data')
OutStatCol=['ID','Name','Longitude','Latitude','GRS','OriginalVariable','VariableUnits','URL','JsonNode']
s.to_excel(writer, columns=OutStatCol, index=False, sheet_name='Stations')
writer.save()

应该可以

弄清楚 writer 不是从工作簿继承的，我花了比应该花的时间更长的时间。 writer.book 直接就是一个workbook实例...d'oh