Python 网页抓取 Zacks 网站错误 : [WinError 10054] An existing connection was forcibly closed by the remote host

Question

我想获取此页面上的数据: https://www.zacks.com/stock/quote/MA

我尝试使用 Python 中的 Beautiful Soup 执行此操作，但收到错误:“[WinError 10054] 现有连接被远程主机强制关闭”。

有人可以指导我吗？

from bs4 import BeautifulSoup
import urllib
import re
import urllib.request

url = 'https://www.zacks.com/stock/quote/MA'

r = urllib.request.urlopen(url).read()
soup = BeautifulSoup(r, "lxml")
soup

谢谢!

Answer 1

该网站正在阻止您的请求，可能主机不允许没有请求 header 的请求。您可以尝试使用 Selenium 包模拟“真实”请求。

这正在工作:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup




options = Options()

options.set_headless(headless=True)

url = 'https://www.zacks.com/stock/quote/MA'

browser = webdriver.Firefox(firefox_options=options)

browser.get(url)

html_source = browser.page_source

soup = BeautifulSoup(html_source, "lxml")

print(soup)

browser.close()