python - 如何使用 BeautifulSoup 将平面 HTML 结构解析为字典？

Question

我正在尝试使用 BeautifulSoup 解析来自具有平面 HTML 结构的体育网站的赛程数据。

到目前为止，我尝试只解析每个赛程日期的第一个赛程，而不解析同一日期的其他赛程。

html 是:

...

<h3 class="fix_header1">January 2019</h3>
<h4 class="fix_header2">Friday 4th January</h4>
<div class="fix_item">
    <span class="match_col">
        <span class="team">Warriors</span>
    </span>
    <span class="match_time">20:00</span>
    <span class="match_col">
        <span class="team">Knights</span>
    </span>
</div>
<h4 class="fix_header2">Saturday 5th January</h4>
<div class="fix_item">...</div>
<div class="fix_item">...</div>
<div class="fix_item">...</div>
<h4 class="fix_header2">Sunday 6th January</h4>
<div class="fix_item">...</div>
<div class="fix_item">...</div>
<div class="fix_item">...</div>
<div class="fix_item">...</div>
<div class="fix_item">...</div>
<div class="fix_item">...</div>

...

代码:

from bs4 import BeautifulSoup
import requests

url = "https://www.dummyurl.com/fixtures"
response = requests.get(url, timeout=5)
content = BeautifulSoup(response.content, "html.parser")

fixtures = []

def process_fixtures(date, home, time, away):
    fixture_item = {
        "date": "", 
        "home":"", 
        "time":"", 
        "away":""
    }

    fixture_item["date"] = date
    fixture_item["home"] = home
    fixture_item["time"] = time
    fixture_item["away"] = away
    fixtures.append(fixture_item)

fixtures_dates = content.find_all("h4", class_="fix_header2")
for fixtures_date in fixtures_dates:
    date = fixtures_date.text 
    home = fixtures_date.find_next("span", class_="team").text
    time = fixtures_date.find_next("span", class_="match_time").text.strip()
    away = fixtures_date.find_next("span", class_="team").find_next("span", class_="team").text
    process_fixtures(date, home, time, away)

输出:

[{'date': 'Friday 4th January',
  'home': 'Warriors',
  'time': '20:00',
  'away': 'Knights'},
 {'date': 'Saturday 5th January',
  'home': 'Kings',
  'time': '15:00',
  'away': 'Bulls'},
 {'date': 'Sunday 6th January',
  'home': 'Fishes',
  'time': '19:00',
  'away': 'Lions'}, 

  ...

我在寻找什么:

[{'date': 'Friday 4th January',
  'home': 'Warriors',
  'time': '20:00',
  'away': 'Knights'},
 {'date': 'Saturday 5th January',
  'home': 'Kings',
  'time': '15:00',
  'away': 'Bulls'},
 {'date': 'Saturday 5th January',
  'home': 'Cats',
  'time': '16:30',
  'away': 'Dogs'},
 {'date': 'Saturday 5th January',
  'home': 'Empire',
  'time': '19:30',
  'away': 'County State'},

   ...

Answer 1

技巧是从日期标题开始，然后循环包含夹具信息的同级标题，直到您点击另一个标题。您在日期标题之间收集的任何内容都属于最后一个日期。

试试这个:

from bs4 import BeautifulSoup, Tag
import requests
from pprint import pprint

def make_soup(url: str) -> BeautifulSoup:
    res = requests.get(url, headers={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0'})
    res.raise_for_status()
    html = res.text
    soup = BeautifulSoup(html, 'html.parser')
    return soup

def extract_fixtures(soup: BeautifulSoup) -> list:
    headers = soup.select('.fixres__header2')
    fixtures = []
    for h in headers:
        date = h.text.strip()
        for s in h.next_siblings:
            if s in headers:
                break
            if not isinstance(s, Tag):
                continue
            if 'fixres__item' not in s.get('class', []):
                break

            home = s.select_one('.matches__participant--side1').text.strip()
            away = s.select_one('.matches__participant--side2').text.strip()
            time = s.select_one('.matches__date').text.strip()
            m = {
                'date': date,
                'home': home,
                'away': away,
                'time': time
            }
            fixtures.append(m)
    return fixtures


url = 'https://www.skysports.com/premier-league-fixtures'
soup = make_soup(url)
fix = extract_fixtures(soup)

pprint(fix)

输出:

[{'away': 'Norwich City',
  'date': 'Friday 9th August',
  'home': 'Liverpool',
  'time': '20:00'},
 {'away': 'Manchester City',
  'date': 'Saturday 10th August',
  'home': 'West Ham United',
  'time': '12:30'},
 {'away': 'Sheffield United',
  'date': 'Saturday 10th August',
  'home': 'Bournemouth',
  'time': '15:00'},
 {'away': 'Southampton',
  'date': 'Saturday 10th August',
  'home': 'Burnley',
  'time': '15:00'},
 {'away': 'Everton',
  'date': 'Saturday 10th August',
  'home': 'Crystal Palace',
  'time': '15:00'},
 {'away': 'Brighton and Hove Albion',
  'date': 'Saturday 10th August',
  'home': 'Watford',
  'time': '15:00'},
...
...