ruby-on-rails - 如何使用 Nokogiri 解析 HTML 表格？

Question

我正在尝试解析一个表，但我不知道如何从中保存数据。我想将每一行中的数据保存为:

['Raw name 1', 2,094, 0,017, 0,098, 0,113, 0,452]

示例表是:

html = <<EOT
    <table class="open">
        <tr>
            <th>Table name</th>
            <th>Column name 1</th>
            <th>Column name 2</th>
            <th>Column name 3</th>
            <th>Column name 4</th>
            <th>Column name 5</th>
        </tr>
        <tr>
            <th>Raw name 1</th>
            <td>2,094</td>
            <td>0,017</td>
            <td>0,098</td>
            <td>0,113</td>
            <td>0,452</td>         
        </tr>
        .
        .
        .
        <tr>
            <th>Raw name 5</th>
            <td>2,094</td>
            <td>0,017</td>
            <td>0,098</td>
            <td>0,113</td>
            <td>0,452</td>         
        </tr>
    </table>
EOT

我的爬虫代码是:

  doc = Nokogiri::HTML(open(html), nil, 'UTF-8')
  tables = doc.css('div.open')

  @tablesArray = []

  tables.each do |table|
    title = table.css('tr[1] > th').text
    cell_data = table.css('tr > td').text
    raw_name = table.css('tr > th').text
    @tablesArray << Table.new(cell_data, raw_name)
  end

  render template: 'scrape_krasecology'
  end
  end

当我尝试在 HTML 页面中显示数据时，看起来所有列名都存储在一个数组的元素中，所有数据都以相同的方式存储。

Answer 1

问题的关键是对多个结果调用#text 将返回每个单独元素的#text 的串联。

让我们检查一下每个步骤的作用:

# Finds all <table>s with class open
# I'm assuming you have only one <table> so
#  you don't actually have to loop through
#  all tables, instead you can just operate
#  on the first one. If that is not the case,
#  you can use a loop the way you did
tables = doc.css('table.open')

# The text of all <th>s in <tr> one in the table
title = table.css('tr[1] > th').text

# The text of all <td>s in all <tr>s in the table
# You obviously wanted just the <td>s in one <tr>
cell_data = table.css('tr > td').text

# The text of all <th>s in all <tr>s in the table
# You obviously wanted just the <th>s in one <tr>
raw_name = table.css('tr > th').text

---

现在我们知道出了什么问题，下面是一个可能的解决方案:

html = <<EOT
    <table class="open">
        <tr>
            <th>Table name</th>
            <th>Column name 1</th>
            <th>Column name 2</th>
            <th>Column name 3</th>
            <th>Column name 4</th>
            <th>Column name 5</th>
        </tr>
        <tr>
            <th>Raw name 1</th>
            <td>1001</td>
            <td>1002</td>
            <td>1003</td>
            <td>1004</td>
            <td>1005</td>         
        </tr>
        <tr>
            <th>Raw name 2</th>
            <td>2001</td>
            <td>2002</td>
            <td>2003</td>
            <td>2004</td>
            <td>2005</td>         
        </tr>
        <tr>
            <th>Raw name 3</th>
            <td>3001</td>
            <td>3002</td>
            <td>3003</td>
            <td>3004</td>
            <td>3005</td>         
        </tr>
    </table>
EOT

---

doc = Nokogiri::HTML(html, nil, 'UTF-8')

# Fetches only the first <table>. If you have
#  more than one, you can loop the way you
#  originally did.
table = doc.css('table.open').first

# Fetches all rows (<tr>s)
rows = table.css('tr')

# The column names are the first row (shift returns
#  the first element and removes it from the array).
# On that row we get the text of each individual <th>
# This will be Table name, Column name 1, Column name 2...
column_names = rows.shift.css('th').map(&:text)

# On each of the remaining rows
text_all_rows = rows.map do |row|

  # We get the name (<th>)
  # On the first row this will be Raw name 1
  #  on the second - Raw name 2, etc.
  row_name = row.css('th').text

  # We get the text of each individual value (<td>)
  # On the first row this will be 1001, 1002, 1003...
  #  on the second - 2001, 2002, 2003... etc
  row_values = row.css('td').map(&:text)

  # We map the name, followed by all the values
  [row_name, *row_values]
end

p column_names  # => ["Table name", "Column name 1", "Column name 2",
                #     "Column name 3", "Column name 4", "Column name 5"]
p text_all_rows # => [["Raw name 1", "1001", "1002", "1003", "1004", "1005"],
                #     ["Raw name 2", "2001", "2002", "2003", "2004", "2005"],
                #     ["Raw name 3", "3001", "3002", "3003", "3004", "3005"]]

# If you want to combine them
text_all_rows.each do |row_as_text|
  p column_names.zip(row_as_text).to_h
end # =>
    # {"Table name"=>"Raw name 1", "Column name 1"=>"1001", "Column name 2"=>"1002", "Column name 3"=>"1003", "Column name 4"=>"1004", "Column name 5"=>"1005"}
    # {"Table name"=>"Raw name 2", "Column name 1"=>"2001", "Column name 2"=>"2002", "Column name 3"=>"2003", "Column name 4"=>"2004", "Column name 5"=>"2005"}
    # {"Table name"=>"Raw name 3", "Column name 1"=>"3001", "Column name 2"=>"3002", "Column name 3"=>"3003", "Column name 4"=>"3004", "Column name 5"=>"3005"}