python - 网页抓取 | BeautifulSoup |解析表

Question

这方面有一些很棒的线程(其中一些帮助我达到了这一点)，但我似乎无法弄清楚为什么我的程序无法正常工作。

问题:程序可以工作，但它似乎只返回第一行，而它应该循环遍历所有表行。

我使用的是Python 3.5

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "http://www.the-numbers.com/movies/year/2006"

r = requests.get(url)
soup = BeautifulSoup(r.content)

data = []

for table_row in soup.select("table"):
    cells = table_row.find_all(['td'])
    release_date = cells[0].text.strip()
    movie_name = cells[2].text.strip()
    genre_name = cells[3].text.strip()
    production_budget = cells[4].text.strip()
    box_office = cells[5].text.strip()
    movie = {"Release_Date" : release_date, 
             "Movie_Name" : movie_name, 
             "Genre" : genre_name, 
             "Production_Budget" : production_budget, 
             "Box_Office" : box_office}
    data.append(movie)
    print (release_date, movie_name, genre_name, production_budget, box_office)

这会返回正确的 2006 年 1 月 BloodRayne Action $25,000,000 $2,405,420，但我需要表中的所有其他行。

如果这个问题很容易解决，下一步就是将其放入 Pandas DataFrame 中(但在响应中不是必需的)。

任何帮助将不胜感激。

Answer 1

您可以使用read_html进行一些数据清理:

df = pd.read_html('http://www.the-numbers.com/movies/year/2006', header=0)[0]
df = df.dropna(how='all')
df['Release Date'] = df['Release Date'].ffill()
print (df.head())
    Release Date          Movie   Genre ProductionBudget  \
0  January, 2006            NaN     NaN              NaN   
1      January 6     BloodRayne  Action      $25,000,000   
2      January 6       Fateless   Drama      $12,000,000   
3      January 6  Grandma's Boy  Comedy       $5,000,000   
4      January 6         Hostel  Horror       $4,800,000   

  DomesticBox Officeto Date  Trailer  
0                       NaN      NaN  
1                $2,405,420      NaN  
2                  $196,857      NaN  
3                $6,090,172      NaN  
4               $47,326,473      NaN  

您原来的解决方案:

data = []
#find first table
tab = soup.select("table")[0]
#find all tr elements
rows = tab.find_all(['tr'])
#loop anf find all td
for row in rows:
    cols = row.find_all('td')
    #parse text
    cols = [ele.text.strip() for ele in cols]
    #[:-1] remove last column
    data.append(cols[:-1])

cols = ['Release_Date','Movie_Name','Genre','Production_Budget','DomesticBox']
#[2:] remove first 2 rows
df = pd.DataFrame(data[2:], columns = cols)
print (df.head())
  Release_Date     Movie_Name   Genre Production_Budget  DomesticBox
0    January 6     BloodRayne  Action       $25,000,000   $2,405,420
1                    Fateless   Drama       $12,000,000     $196,857
2               Grandma's Boy  Comedy        $5,000,000   $6,090,172
3                      Hostel  Horror        $4,800,000  $47,326,473
4               Kill the Poor                                     $0