python - 使用 python 进行网页抓取 - 较旧的 html 设计。 .数据表

Question

我正在尝试从 this 获取信息地点。我本质上想从网站上获取所有可以获取的信息。所以整个 table 就很理想了。 html 中有一个 .DataTable 元素——我认为它应该很容易抓取。我在提取数据时遇到问题。

我首先尝试了一个简单的pandas read_html:

link = link
tables = pd.read_html(link,header=0)[0]
tables

这不起作用，我认为这是因为该网站实际上不是一个表格。所以我尝试使用 BeautifulSoup，我认为这不会有问题:

import requests
from bs4 import BeautifulSoup

res = requests.get(link)
soup = BeautifulSoup(res.content, "html.parser")
tables = soup.find(".DataTable")

它不起作用。我认为会是因为 .DataTable 是我使用 SelectorGadget 时的元素。

我还尝试了以下方法:

res = requests.get(link)
soup = BeautifulSoup(res.content, "html.parser")
trs = soup.find("div", attrs={"class": "DataTable"}).find("table").findAll("tr")
for tr in trs:
    tds = tr.findAll("td")
    if len(tds) < 1:
        continue
    name = tds[0].text
    print(name)
f.close()

这就是我开始脱离我的元素的地方。我不确定需要拉动什么杠杆才能使上述工作发挥作用。

接下来我应该尝试什么？有什么建议吗？

Answer 1

您可以从这个脚本开始。它对 .DataTable 进行一些基本的抓取:

import requests
from bs4 import BeautifulSoup

url = 'https://www.pluginillinois.org/OffersBegin.aspx'

with requests.session() as s:
    soup = BeautifulSoup(s.get(url).text, 'lxml')

    post_data = {i['name']: i['value'] for i in soup.select('input[name][value]')}

    soup = BeautifulSoup(s.post(url, data=post_data).text, 'lxml')

    data = []
    for row in soup.select('.DataTable tr:has(td)'):
        data += [ [td.get_text(separator=', ', strip=True).replace(', Compare', '') for td in row.select('td')] ]


from textwrap import shorten
for row in data:
    for i, d in enumerate(row):
        if i in (1, 2, 3, 4, 5, 6):
            print('{: ^16}'.format(shorten(str(d[0]) if len(d) == 1 else str(d), 16)), end='|')
        else:
            print('{: <45}'.format(shorten(str(d[0]) if len(d) == 1 else str(d), 45)), end='|')
    print()

打印:

1-800-755-5000                               |  Fixed [...]   |       $0       |     $25.19     |     $49.06     |     $70.96     |                |This Price to Compare is in effect from [...]|
877.808.1022, Three months fixed at [...]    |  Fixed [...]   |       0        |     $20.75     |     $41.50     |     $62.25     |     3, $0      |A low fixed electricity rate of 4.15 [...]   |
888-354-4415, 12 Month Fixed                 |  Fixed [...]   |       $0       |     $28.40     |     $56.80     |     $85.20     |    12, $50     |An affordable 12 month fixed rate plan [...] |
(877) 282-6248, Illinois Select Variable Plan| Variable [...] |       $0       |     $38.50     |     $77.00     |    $115.50     |     0, $0      |Illinois Select Variable (Rate Zone I): [...]|
1-866-587-8674, 2-Year Price Protection Plan |  Fixed [...]   |       $0       |     $39.95     |     $79.90     |    $119.85     |    24, $50     |Lock in your electricity rate 24 months [...]|
1-866-769-3799, Fixed IL 24                  |  Fixed [...]   |       0        |                |                |                |    24, $50     |Our customer loyalty program is offered [...]|
AMEREN_FCBB_08                               |  Fixed [...]   |       0        |     $30.00     |     $60.00     |     $90.00     |      6, 0      |6 month fixed With a $50 Cash Back Bonus     |
855-243-7270, Worry Free 24-month fixed      |  Fixed [...]   |       $0       |     $39.67     |     $79.33     |    $119.00     |    24, $50     |Lock-in now for a worry free [...]           |
866-663-2508, Value Fixed                    |  Fixed [...]   |       $0       |     $36.45     |     $72.90     |    $109.35     |    24, $50     |Value Fixed for 24 months!                   |
888-410-5081, 12-Month Green Fixed Rate      |  Fixed [...]   |       $0       |     $40.58     |     $81.16     |    $121.74     |    12, $49     |Ilinois RPS requires that 14.5% of our [...] |
1-888-504-6372, Indra Web Only - Ameren [...]|  Fixed [...]   |       0        |     $44.50     |     $89.00     |    $133.50     |     18, 0      |18 month fixed term contract with [...]      |
(800)654-6990, GreenValueAssurance6Plus      | Variable [...] |     $9.99      |     $25.45     |     $50.89     |     $76.35     |      6, 0      |GreenValueAssurancePlus is a 6 [...]         |
866-663-2508, Value Fixed                    |  Fixed [...]   |       $0       |     $36.95     |     $73.90     |    $110.85     |    12, $50     |Value Fixed for 12 months!                   |
1-866-577-4700, 12 Month Plan                |  Fixed [...]   |       $0       |     $27.95     |     $55.90     |     $83.85     |    12, $50     |You will receive a fixed price of 5.59¢ [...]|
844-684-5506, Fixed                          |  Fixed [...]   |       $0       |     $36.95     |     $73.90     |    $110.85     |    12, $50     |12 Month Fixed Rate with Price Certainty.    |
1-855-331-0022, Live Brighter 12             |  Fixed [...]   |       $0       |     $29.45     |     $58.90     |     $88.35     |     12, $0     |A fixed electricity rate for 12 months [...] |
(877) 282-6248, Illinois Select 12 [...]     |  Fixed [...]   |       $0       |     $24.45     |     $48.90     |     $73.35     |     12, $0     |Illinois Select 12 Month Term (Rate [...]    |
888-504-6372, Indra Web Only - Ameren [...]  |  Fixed [...]   |       0        |     $35.50     |     $71.00     |    $106.50     |     18, 0      |Web Exclusive Fixed Green Plan: Initial [...]|
1-844-687-4519, Live Brighter 18             |  Fixed [...]   |       $0       |     $29.95     |     $59.90     |     $89.85     |     18, $0     |18 months of fixed-rate electricity! To [...]|
888-410-5081, 6-Month Fixed Rate             |  Fixed [...]   |       $0       |     $29.99     |     $59.99     |     $89.98     |     6, $49     |We lock in your rate for 6 months. No [...]  |
1-888-828-7374, Locked Ameren Res 12 mo [...]|  Fixed [...]   |       0        |     $25.20     |     $50.40     |     $75.60     |12, $10.00 [...]|Residents Energy ResiSURE: Lock in a [...]   |
(800)654-6990, GreenValueAssurance           | Variable [...] |     $9.99      |     $31.45     |     $62.90     |     $94.35     |      5, 0      |GreenValueAssurance is a 5 month,100% [...]  |
1-866-587-8674, Perks Points Plan            |  Fixed [...]   |       $0       |                |                |                |    24, $50     |Sign up and receive up to 5,000 Just [...]   |
1-888-828-7374, Locked Ameren Res 6 mo [...] |  Fixed [...]   |       0        |     $24.85     |     $49.70     |     $74.55     |6, $10.00 [...] |Residents Energy ResiSURE: Lock in a [...]   |
877-300-6747, Illinois ManagedPriceTM - [...]| Variable [...] |     $3.75      |     $25.55     |     $47.35     |     $69.15     |    24, $50     |The Illinois ManagedPriceTM is a [...]       |
888-410-5081, 3-Month Fixed Rate             |  Fixed [...]   |       $0       |     $24.95     |     $49.90     |     $74.85     |     3, $49     |We lock in your rate for 3 months. No [...]  |
1-888-504-6372, Indra Web Only - Ameren [...]|  Fixed [...]   |       0        |     $33.00     |     $66.00     |     $99.00     |     14, 0      |Web Exclusive Fixed Plan: Initial rate [...] |
877-648-1927, AEP Energy's Traditional [...] |  Fixed [...]   |       0        |     $21.95     |     $43.90     |     $65.85     |      6, 0      |Includes access to AEP Energy Reward [...]   |
(877) 648-1927, AEP Energy's [...]           |  Fixed [...]   |       $0       |     $22.45     |     $44.90     |     $67.35     |     12, $0     |Includes access to AEP Energy Reward [...]   |
844.692.4372, 12-Month Fixed - Agera [...]   |  Fixed [...]   |       $0       |     $38.94     |     $77.88     |    $116.82     |    12, $50     |The Agera Fixed Advantage gives you [...]    |
855-388-5276, Cash Back Electric Plan        |  Fixed [...]   |       $0       |     $25.50     |     $51.00     |     $76.50     |    6, $0.00    |Get $50 sign up bonus after 6 months of [...]|
1-866-769-3799, Independence 18              |  Fixed [...]   |       0        |                |                |                |    18, $50     |Our customer loyalty program is offered [...]|
(877) 282-6248, Illinois Select Green [...]  | Variable [...] |       $0       |     $48.60     |     $97.20     |    $145.80     |     0, $0      |Illinois Select Green Variable (Rate [...]   |
1-888-504-6372, Indra Web Only - Ameren [...]|  Fixed [...]   |       0        |     $41.00     |     $82.00     |    $123.00     |     14, 0      |Web Exclusive 14 month fixed contract: [...] |
855-243-7270, Worry Free 12-month fixed      |  Fixed [...]   |       $0       |     $39.29     |     $78.57     |    $117.86     |    12, $50     |Lock-in now for a worry free [...]           |
(877) 648-1927, AEP Energy's Renewable [...] |  Fixed [...]   |       $0       |     $22.95     |     $45.90     |     $68.85     |     12, $0     |Includes access to AEP Energy Reward [...]   |
1-800-654-6990, ClearCharge12 EV             |  Fixed [...]   |       0        |     $34.45     |     $68.90     |    $103.35     |    12, $50     |ClearCharge12 EV is a 12 month,100% [...]    |
1-866-577-4700, 12 Month Plan - 100% Green   |  Fixed [...]   |       $0       |     $28.45     |     $56.90     |     $85.35     |    12, $50     |Receive a fixed price of 5.69¢ per kWh [...] |
Go Green Lights 24                           |  Fixed [...]   |       0        |     $30.95     |     $61.90     |     $92.85     |     24, 0      |Reducing your carbon footprint is [...]      |
(877) 282-6248, Illinois Select Green Term   |  Fixed [...]   |       $0       |     $34.45     |     $68.90     |    $103.35     |     12, $0     |Illinois Select Green 12 Month Term [...]    |
888-354-4415, 24 Month Fixed                 |  Fixed [...]   |       $0       |     $29.30     |     $58.60     |     $87.90     |    24, $50     |An affordable 24 month fixed rate plan [...] |
855-388-5276, NRG 24M Price Lock Plan        |  Fixed [...]   |       $0       |     $32.00     |     $64.00     |     $96.00     |   24, $0.00    |The Illinois Price Lock Plan includes [...]  |
1-866-769-3799, Fixed IL 18                  |  Fixed [...]   |       0        |                |                |                |    18, $50     |Our customer loyalty program is offered [...]|
844.692.4372, 24-Month Fixed - Agera [...]   |  Fixed [...]   |       0        |     $39.28     |     $78.56     |    $117.84     |     24, 50     |The Agera Fixed Advantage gives you [...]    |
1-800-654-6990, ClearGreenGuarantee 12       |  Fixed [...]   |      None      |     $31.95     |     $63.90     |     $95.85     |    12, $50     |ClearGreenGuarantee 12 is a 12 [...]         |
1-866-577-4700, 18 Month Plan                |  Fixed [...]   |       $0       |     $28.45     |     $56.90     |     $85.35     |    18, $50     |Receive a fixed price of 5.69¢ per kWh [...] |
Power on Command 24                          |  Fixed [...]   |       0        |     $32.45     |     $64.90     |     $97.35     |     24, 0      |Lock in a competitive fixed electricity [...]|
8667057291, 12M Fixed Residential            |  Fixed [...]   |       $0       |     $40.45     |     $80.90     |    $121.35     |   12, $50.00   |Our fixed rate program gives consumers [...] |
855-388-5276, Renewable Electric Plan        |  Fixed [...]   |       $0       |     $26.50     |     $53.00     |     $79.50     |    6, $0.00    |100% National Wind Energy! Plus, get a [...] |
(800)654-6990, GreenValueAssurancePlus       | Variable [...] |       0        |     $27.95     |     $55.90     |     $83.85     |      6, 0      |GreenValueAssurancePlus is a 6 [...]         |
1-855-331-0022, Live Brighter 17             |  Fixed [...]   |       0        |     $27.45     |     $54.90     |     $82.35     |     17, 0      |A fixed electricity rate for 10 months [...] |