python - 使用 python 抓取谷歌精选片段

Question

https://www.google.com/search?q=LAPTOP+ACER+I3/4/1TB/8GEN+full+specs

例如:我想搜索该产品并直接从特色片段中抓取它的规范。我怎样才能把所有东西都放进那个盒子里？？

Answer 1

根据 Google featured snippets ,

Featured snippets come from web search listings. Google's automated systems determine whether a page would make a good featured snippet to highlight for a specific search request.

因此，如果您想抓取多个搜索，这将不是一种可靠的方法，因为它们会千差万别。

但是，对于这个特定的搜索，您可以重定向您的抓取工具以跟踪该链接，然后您必须编写代码来抓取该链接的信息。

How do I get everything inside of that box??

该框只包含您可以看到的信息，可能不是您想要的所有信息。如果您只想抓取该信息，那非常简单。

import requests
from bs4 import BeautifulSoup

response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
div = soup.find("div", {"class": "webanswers-webanswers_table__webanswers- 
table"})
tr = div.findAll("tr")
for row in tr:
    td = row.findAll("td")
    print(td[0].text.strip(), ": " ,td[1].text.strip())

如果上述代码不起作用或返回 429 或其他状态代码，Google 可能会阻止抓取脚本/蜘蛛程序。尝试添加用户代理，例如:

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36'}
response = requests.get(url, headers=headers)

如果同样失败，请尝试使用 selenium

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait as wait

url = "https://www.google.com/search?q=acer+i3+8th+gen+1tb+laptop+full+specs"

driver = webdriver.Firefox("path/to/geckodriver")
driver.get(url)

snippet = wait(driver, 60).until(lambda driver: 
driver.find_element_by_css_selector("div.webanswers-webanswers_table__webanswers-table"))
print(snippet.text)