Python cElementTree 使用 2GB RAM 迭代解析大文件

Question

我正在尝试迭代解析一个大型 (186MB) XML 文件。文件本身只是复杂的 MyNodeType 节点的列表。例如，

<MyNodeTypeList>
    <MyNodeType>
        ...
    </MyNodeType>
    <MyNodeType>
        ...
    </MyNodeType>
    <MyNodeType>
        ...
    </MyNodeType>
</MyNodeTypeList>

我试图通过迭代解析文件来保持较低的内存使用量，但它似乎在 30k 记录左右大幅减慢，并在 92k 处停止。即使使用如下所示的简单代码，该进程也会占用大约 2GB 的内存:

import xml.etree.cElementTree as ET

def main(argv):
    it = ET.iterparse(argv[0])
    count = 0
    for (ev, el) in it:
        if (ev == "end" and el.tag == "MyNodeType"):
            count += 1
    print count

if __name__ == "__main__":
   main(sys.argv[1:])

有什么方法可以将处理代码的内存使用量保持在最低限度吗？

Answer 1

iterparse 的这种行为是 documented :

Note that iterparse still builds a tree, just like parse, but you can safely rearrange or remove parts of the tree while parsing

为了避免将整个树保留在内存中，请使用示例中提供的方法:

# get an iterable
context = iterparse(source, events=("start", "end"))

# turn it into an iterator
context = iter(context)

# get the root element
event, root = context.next()

for event, elem in context:
    if event == "end" and elem.tag == "record":
        ... process record elements ...
        root.clear()