ruby - Nokogiri::XML::Reader - 处理大型 XML 文件并跳过不感兴趣的节点

Question

我有一些格式如下所示的 xml，我正在尝试使用 Nokogiri::XML::Reader 进行解析，因为文件大小非常大(~1GB)。该文件有很多packets以下格式。

来自每个packet我需要收集frame.time_epoch , s1ap.procedureCode .

我目前正在做以下事情。

data = []
file = `some_file.xml`
reader = Nokogiri::XML::Reader(File.open(file))
reader.each do |node|
    if (node.name == 'packet' && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT)
      doc = Nokogiri::XML(node.outer_xml)
      next if !doc.css("field[name='s1ap.procedureCode']") ## do nothing if the <packet> is not of s1ap type
      epochTime = doc.css("field[name='frame.time_epoch']").first["show"].to_i
      procedureCode = procedureCode_node = doc.css("field[name='s1ap.procedureCode']").first["show"].to_i
      data << { epochTime: epochTime, procedureCode: procedureCode }
    end
end

问题

我面临的挑战是解析速度非常慢。我注意到的一件事是读者会扫描 <packet> </packet> 中的所有后续行。 - 有没有办法让读者移动到名称为 packet 的下一个节点？而是遍历 packet 中的每一行进一步。

XML 格式

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267259.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="s1ap.procedureCode" showname="procedureCode: id-downlinkNASTransport (11)" size="1" pos="79" show="11" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267260.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="Some other protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="other.OTH_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="other.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="other.procedureCode" showname="procedureCode: id-someTransport (99)" size="1" pos="79" show="11" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<packet>
  <proto name="geninfo" pos="0" showname="General information" size="126">
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/>
  </proto>
  <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0">
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/>
    <field name="frame.time_epoch" showname="Epoch Time: 1474267261.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/>
  </proto>
  <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0">
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591">
      <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/>
    </field>
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/>
  </proto>  
  <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78">
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/>
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1">
      <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value="">
        <field name="s1ap.procedureCode" showname="procedureCode: id-uplinkTransport (13)" size="1" pos="79" show="13" value="0b"/>
       </field>
    </field>
  </proto>
</packet>
<!-- more <packet>s here -->

Answer 1

对于这么大的文档，您应该使用 SAX 解析器

http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/SAX

流式处理大型文档而不将整个文档拉入内存并将其作为 DOM 进行解析是更可取的。特别是考虑到这个问题只需要一次通过。

这是通过使用 SAX 流式传输 XML 来完成任务的代码:

require 'nokogiri'

class PacketFilter < Nokogiri::XML::SAX::Document
  def initialize
    reset
  end

  def end_document
    puts 'the document has ended'
  end

  def start_element(name, attributes = [])
    case name
    when 'packet'
      @in_packet = true
    when 'proto'
      @have_s1ap = @in_packet && attribute_value(attributes, 'name') == 's1ap'
    when 'field'
      case attribute_value(attributes, 'name')
      when 's1ap.procedureCode'
        @procedure_code = attribute_value(attributes, 'showname')
      when 'frame.time_epoch'
        @epoch_time = attribute_value(attributes, 'showname')
      end
    end
  end

  def end_element(name)
    if name == 'packet'
      puts "#{@procedure_code}, #{@epoch_time}" if @have_s1ap
      reset
    end
  end

  private

  def attribute_value(attributes, name)
    attributes.reduce(nil) do |value, assoc|
      assoc[0] == name ? assoc[1] : value
    end
  end

  def reset
    @in_packet = false
    @have_s1ap = false
    @procedure_code = nil
    @epoch_time = nil
  end
end

parser = Nokogiri::XML::SAX::Parser.new(PacketFilter.new)
parser.parse($stdin)

如果您将数据示例粘贴到 data.xml 并将上面的 ruby​​ 粘贴到 slap.rb:

$ cat data.xml | ruby poke.rb
procedureCode: id-downlinkNASTransport (11), Epoch Time: 1474267259.184197000 seconds
the document has ended