python - 在 Python 中使用 BeautifulSoup 从 HTML 文本中的嵌套元素中获取文本

Question

我正在尝试提取每天参加比赛的球队以及每支球队阵容中的活跃和不活跃球员。我要抓取的页面的 URL 是:https://stats.nba.com/lineups/ .我一直在使用 BeautifulSoup 来尝试获取这些数据，并尝试了几种方法来获取它，但我似乎无法在

<div class=​"landing__flex-col lineups-game" data-game-state=​"3" nba-data-game=​"game" nba-with ng-include ng-repeat=​"game in games" src=​"'/​lineups-template.html'">​ .

我想在每场比赛中获得每场比赛的球队

<div class=​"landing__flex-col lineups-game" data-game-state=​"3" nba-data-game=​"game" nba-with ng-include ng-repeat=​"game in games" src=​"'/​lineups-template.html'">​ ,

和

中的每个玩家

<div class=​"columns small-6 lineups-game__team lineups-game__team--htm" nba-with nba-with-data-team=​"game.h" ng-include src=​"'/​lineups-team-template.html'">​ .

因此，在下面的 html 代码示例中，我想获取 MEM、CHA、J. Valanciunas 和 J. Crowder 的文本，并最终为每支球队的每位球员执行此操作。

<div class="landing__flex-row lineups-games" ng-show="isLoaded &amp;&amp; hasData" aria-hidden="false">
          <!----><!----><div class="landing__flex-col lineups-game" ng-repeat="game in games" nba-with="" nba-data-game="game" data-game-state="3" ng-include="" src="'/lineups-template.html'">
  <div class="lineups-game__inner row">

    <div class="columns small-12 lineups-game__title">
      <a href="/game/0021900154/">
        <span class="lineups-game__team-name">MEM</span>
        <span class="lineups-game__vs">vs</span>
        <span class="lineups-game__team-name">CHA</span>
        <span class="lineups-game__status hide-for-live-game">Final</span>
        <span class="lineups-game__status hide-for-pre-game hide-for-post-game">Live</span>
      </a>
    </div>

    <!----><div class="columns small-6 lineups-game__team lineups-game__team--vtm" nba-with="" nba-with-data-team="game.v" ng-include="" src="'/lineups-team-template.html'">

  <!----><!----><div ng-if="team.hasBench" nba-with="" nba-with-data-team="team" ng-include="" src="'/lineups-confirmed-roster-template.html'">
  <div class="lineups-game__header">
    <img team-logo="" class="lineups-game__team-logo team-img" abbr="MEM" type="image/svg+xml" src="/media/img/teams/logos/MEM_logo.svg" alt="Memphis Grizzlies logo" title="Memphis Grizzlies logo">
    <span class="lineups-game__team-name">MEM</span>
  </div>

  <div class="lineups-game__roster-type lineups-game__roster-type--confirmed">Active List</div>

  <ul class="lineups-game__roster lineups-game__roster--official">
    <!----><li class="lineups-game__player lineups-game__player--starter" ng-repeat="pl in team.starters">
      <a href="/player/202685/">
        <span class="lineups-game__pos">C</span>
        <span class="lineups-game__name">J. Valanciunas</span>
      </a>
    </li><!----><li class="lineups-game__player lineups-game__player--starter" ng-repeat="pl in team.starters">
      <a href="/player/203109/">
        <span class="lineups-game__pos">SF</span>
        <span class="lineups-game__name">J. Crowder</span>
      </a>

我尝试了以下方法，但没有成功:

gamesSource = urllib.request.urlopen('https://stats.nba.com/lineups/').read()
gamesSoup = bs.BeautifulSoup(gamesSource,'html.parser')

teams = gamesSoup.find_all("span",{"class":"lineups-game__teams-name"})

所有返回的都是一个空列表，当我尝试获取特定的“跨度”行时，返回的只是“无”。

让我知道出了什么问题，以及我可以做些什么来访问我试图获取的信息。

谢谢。

Sample of HTML Code

Answer 1

借用已经说过的，因为这个页面是通过 api/js 调用生成的，你将需要使用不同的抓取库。我通常去 Selenium 。下面的代码将拉出所有团队和花名册并将它们放在一起。这段代码中可能有一些怪癖，但我认为它会朝着正确的方向前进:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from datetime import date

desired_link = 'https://stats.nba.com/lineups/'

fire_opts = webdriver.FirefoxOptions()
fire_opts.add_argument("-headless")
fire_path = 'geckodriver.exe'
driver = webdriver.Firefox(options=fire_opts,executable_path=fire_path)
driver.get(desired_link)

team_names_list = driver.find_elements_by_class_name('lineups-game__team-name')
team_names = []
for name in team_names_list:
    team_names.append(name.text)

starting_lineup_list = driver.find_elements_by_class_name('lineups-game__roster--projected')
starting_lineup = []
for lineup in starting_lineup_list:
    starting_lineup.append(lineup.text)

driver.quit()

for teams, players in zip(team_names,starting_lineup):
    print(teams,players)

这应该像这样在页面上输出所有不同的团队:

DET PG D. Rose
SG L. Kennard
SF T. Snell
PF B. Griffin
C A. Drummond

格式可能会更好一些，但您可以将其放入电子表格(或您喜欢的任何内容)中以供您随意使用...