python - Python 解析 300 MB 文件时出现内存错误

Question

我正在用 python 3.5 解析一个 xml 文件(291 MB)

import xmltodict, json

with open('Wikipedia-20160404094133.xml', encoding='utf-8') as xml_file:
    dic_xml = xmltodict.parse(xml_file.read(), encoding='utf-8', xml_attribs=True)

但是我得到了错误:

dic_xml = xmltodict.parse(xml_file.read(), encoding='utf-8', xml_attribs=True)
MemoryError

我该怎么做才能解决这个问题？

Answer 1

查看 this .

"xmltodict is very fast (Expat-based) and has a streaming mode with a small memory footprint, suitable for big XML dumps like Discogs or Wikipedia"

本质上，您需要分块读取文件，而 xmltodict 的“流模式”似乎就是为此而构建的。