python - 为什么BeautifulSoup找不到HTML类？

Question

我正在尝试抓取this website在Python中使用requests和BeautifulSoup:

我想获取 class = "ficha-jogo" 的文章标签内的所有信息。当我运行下面的代码时，x 是一个空列表。

url = "https://globoesporte.globo.com/rs/futebol/brasileirao-serie-a/jogo/25-05-2019/gremio-atletico-mg.ghtml"
r = requests.get(url)     
soup = BeautifulSoup(r.content, "lxml")
x = soup.select(".ficha-jogo")
print(x)

我期望它返回包含在 class = "ficha-jogo" 的文章标签中的所有标签。

Answer 1

This网站链接是动态渲染请求article数据。您应该尝试automation selenium库。它允许您抓取 动态渲染请求(js 或 ajax) 页面数据。

from bs4 import BeautifulSoup
from selenium import webdriver

browser = webdriver.Chrome()
url = "https://globoesporte.globo.com/rs/futebol/brasileirao-serie-a/jogo/25-05-2019/gremio-atletico-mg.ghtml"

browser.get(url)
soup = BeautifulSoup(browser.page_source, 'html.parser')

article = soup.find("article",{"class":"ficha-jogo"})
print(article.text)

操作:

GREPaulo Victor 1GOLLeonardo 6LADPedro Geromel 3ZADRodrigues 38ZAEJuninho Capixaba 29LAEMichel  5VOLMaicon 8VOLJean Pyerre 21MECThaciano 16MECEverton 11ATAAlisson 23ATADiego Tardelli 9ATAAndré 90ATAFelipe Vizeu 10ATACAMVictor 1GOLPatric 2LADLeonardo Silva 3ZADIgor Rabello 16ZAEFábio Santos 6LAEJosé Welison 14VOLNathan 23MECJair 88VOLCazares 10MECGeuvânio 49ATALuan 27MECBruninho 43MECRicardo Oliveira 9ATAChará 8ATARenato GaúchoTécnico4 - 3 - 3Esquema TáticoRodrigo SantanaTécnico4 - 4 - 2Esquema TáticoMostrar ficha completaReservasJúlio César 22GOLLéo Moura 2LADRafael Galhardo 42LADRomulo 13VOLDarlan 37VOLMontoya 20MECVico 15ATAPepê 25ATACleiton 40GOLIago Maidana 19ZADHulk 22LAEAdilson 21VOLVinícius 29MECTerans 20MECAlerrandro 44ATAMaicon 11ATAInformações sobre o jogoArena do GrêmioArena Desportiva

下载适用于 Chrome 浏览器的 selenium Web 驱动程序:

http://chromedriver.chromium.org/downloads

安装 Chrome 浏览器的网络驱动程序:

https://christopher.su/2015/selenium-chromedriver-ubuntu/

Selenium 教程:

https://selenium-python.readthedocs.io/