python - 如何使用循环从表中抓取数据以使用 python 获取所有 td 数据

Question

所以我想从网站上获取一些数据。而且我很难获得数据。我可以得到球员的名字，但目前仅此而已。一直在尝试不同的事情。这是我试图通过的示例代码。请注意，有两个表(每个团队一个)。每个玩家的类从“偶数”到“奇数”或“奇数”到“偶数”交替出现，下面是我的 python 脚本示例 html 文件。我标记了我想要的部分。我也在使用 python 2.7

`<table id="nbaGITeamStats" cellpadding="0" cellspacing="0">
      <thead class="nbaGIClippers">
         <tr>
            <th colspan="17">Los Angeles Clippers (1-0)</th> <!-- I want team name  -->
         </tr>
      </thead>
      <tbody><tr colspan="17">
         <td colspan="17" class="nbaGIBoxCat"><span>field goals</span><span>rebounds</span></td>
      </tr>
      <tr>
     <td class="nbaGITeamHdrStatsNoBord" colspan="1">&nbsp;</td>
     <td class="nbaGITeamHdrStats">pos</td>
     <td class="nbaGITeamHdrStats">min</td>
     <td class="nbaGITeamHdrStats">fgm-a</td>
     <td class="nbaGITeamHdrStats">3pm-a</td>
     <td class="nbaGITeamHdrStats">ftm-a</td>
     <td class="nbaGITeamHdrStats">+/-</td>
     <td class="nbaGITeamHdrStats">off</td>
     <td class="nbaGITeamHdrStats">def</td>
     <td class="nbaGITeamHdrStats">tot</td>
     <td class="nbaGITeamHdrStats">ast</td>
     <td class="nbaGITeamHdrStats">pf</td>
     <td class="nbaGITeamHdrStats">st</td>
     <td class="nbaGITeamHdrStats">to</td>
     <td class="nbaGITeamHdrStats">bs</td>
     <td class="nbaGITeamHdrStats">ba</td>
     <td class="nbaGITeamHdrStats">pts</td>
  </tr>
  <tr class="odd">
     <td id="nbaGIBoxNme" class="b"><a href="/playerfile/paul_pierce/index.html">P. Pierce</a></td> <!-- I want player name  -->
     <td class="nbaGIPosition">F</td> <!-- I want position name  -->
     <td>14:16</td> <!-- I want this  -->
     <td>1-4</td>  <!-- I want this  -->
     <td>1-2</td>  <!-- I want this  -->
     <td>2-2</td>  <!-- I want this  -->
     <td>+12</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>3</td>  <!-- I want this  -->
     <td>2</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>5</td>  <!-- I want this  -->
  </tr>

  <tr class="even">
     <td id="nbaGIBoxNme" class="b"><a href="/playerfile/blake_griffin/index.html">B. Griffin</a></td>  <!-- I want this  -->
     <td class="nbaGIPosition">F</td>  <!-- I want this  -->
     <td>26:19</td>  <!-- I want this  -->
     <td>5-14</td>  <!-- I want this  -->
     <td>0-1</td>  <!-- I want this  -->
     <td>1-1</td>  <!-- I want this  -->
     <td>+14</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>5</td>  <!-- I want this  -->
     <td>5</td>  <!-- I want this  -->
     <td>2</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>11</td>  <!-- I want this  -->
  </tr>
  <tr class="odd">
     <td id="nbaGIBoxNme" class="b"><a href="/playerfile/deandre_jordan/index.html">D. Jordan</a></td>  <!-- I want this  -->
     <td class="nbaGIPosition">C</td>  <!-- I want this  -->
     <td>26:27</td>  <!-- I want this  -->
     <td>6-7</td>  <!-- I want this  -->
     <td>0-0</td>  <!-- I want this  -->
     <td>3-5</td>  <!-- I want this  -->
     <td>+19</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>11</td>  <!-- I want this  -->
     <td>12</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>1</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>2</td>  <!-- I want this  -->
     <td>3</td>  <!-- I want this  -->
     <td>0</td>  <!-- I want this  -->
     <td>15</td>  <!-- I want this  -->
  </tr>
   <!-- And so on it will keep changing class from odd to even, even to odd  -->
    <!-- Also note there are to tables one for each team  -->
   <!--this is he table id>>> <table id="nbaGITeamStats" cellpadding="0" cellspacing="0"> -->`

这很长，但我想举一个类切换的例子，这里是我的 python 脚本，我计划在我真正成功地抓取数据后使用字典来保存数据。

import urllib
import urllib2
from bs4 import BeautifulSoup
import re
gamesForDay = ['/games/20151002/DENLAC/gameinfo.html']
for game in gamesForDay:
   url =  "http://www.nba.com/"+game
   page = urllib2.urlopen(url).read()
   soup = BeautifulSoup(page)
   for tr in soup.find_all('table id="nbaGITeamStats'):
    tds = tr.find_all('td')
    print tds

Answer 1

这是我的解决方案。请注意，我有一个略有不同的 BeautifulSoup 版本，不是来自 bs4 的版本，但逻辑可能不会太离谱。仍在 Python2.7 上(在我的例子中是在 Windows 上)。

您可能需要修复与上面显示的不同的播放器部分的一些细微差别，但我认为您将能够处理该部分:-)

import urllib
import urllib2
# from bs4 import BeautifulSoup
from BeautifulSoup import BeautifulSoup
import re
gamesForDay = ['/games/20151002/DENLAC/gameinfo.html']
for game in gamesForDay:
   url =  "http://www.nba.com/"+game
   page = urllib2.urlopen(url).read()
   soup = BeautifulSoup(page)

   # fetch the tables you are interested in
   tables = soup.findAll(id="nbaGITeamStats")
   for table in tables:
       team_name = table.thead.tr.th.text
       # odd/even class rows (tr)
       rows = [ x for x in table.findAll('tr') if x.get('class',None) in ['odd','even'] ]
       for player in rows:
           # search the row cols based on 'id'
           player_name = player.find('td', attrs={'id':'nbaGIBoxNme'}).text

           # search the row cols based on 'class'
           player_position = player.find('td', attrs={'class':'nbaGIPosition'}).text

           # search for all td where the class is not defined
           player_numbers = [ x.text for x in player.findAll('td', attrs={'class':None})]

           print player_name, player_position, player_numbers

对于 bs4(我了解到的 BeautifulSoup4)，必须进行一些修改。您仍然需要处理一些东西，但这会提取您想要的大部分数据:

import urllib
import urllib2
from bs4 import BeautifulSoup
import re
gamesForDay = ['/games/20151002/DENLAC/gameinfo.html']
for game in gamesForDay:
   url =  "http://www.nba.com/"+game
   page = urllib2.urlopen(url).read()
   soup = BeautifulSoup(page, "html.parser")

   # fetch the tables you are interested in
   tables = soup.findAll(id="nbaGITeamStats")
   for table in tables:
       team_name = table.thead.tr.th.text
       # odd/even class rows (tr)
       rows = table.find_all(attrs={'class':'odd'})
       rows.extend(table.find_all(attrs={'class':'even'}))

       for player in rows:
           # search the row cols based on 'id'
           player_name = player.find('td', attrs={'id':'nbaGIBoxNme'}).text

           # search the row cols based on 'class'
           player_position = player.find('td', attrs={'class':'nbaGIPosition'}).text

           # search for all td where the class is not defined
           player_numbers = [ x.text for x in player.findAll('td', attrs={'class':None})]

           print player_name, player_position, player_numbers