python - 如何修复由于服务器阻止网页抓取而产生的这些错误？

Question

我正在尝试使用“get_text”函数从网页获取文本，如所述 here .

import urllib.request
from inscriptis import get_text

url = "http://www.informationscience.ch"
html = urllib.request.urlopen(url).read().decode('utf-8')

text = get_text(html)

print(text)

这对于这个特定的网站来说效果很好，但是当我尝试从另一个网站抓取时，我收到 403 错误:

import urllib.request
from inscriptis import get_text

url = "https://economictimes.indiatimes.com/markets/stocks/news/birla-group-enters-the-fray-to-acquire-idbi-federal-life/articleshow/64251332.cms"
html = urllib.request.urlopen(url).read().decode('utf-8')

text = get_text(html)

print(text)

这会在 html = urllib.request.urlopen(url).read().decode('utf-8') 行中出现以下错误:

HTTPError: HTTP Error 403: Forbidden

<小时/>

我尝试通过指定用户代理来修复它，如下所示:

import urllib.request
from inscriptis import get_text

url = "https://economictimes.indiatimes.com/markets/stocks/news/birla-group-enters-the-fray-to-acquire-idbi-federal-life/articleshow/64251332.cms"
html = urllib.request.urlopen(url, headers={'User-Agent': 'Mozilla/5.0'}).read().decode('utf-8')

text = get_text(html)

print(text)

但我收到以下错误:

TypeError: urlopen() got an unexpected keyword argument 'headers'

<小时/>

由于错误提示 headers 未定义 urlopen，因此我尝试使用 requests 模块指定用户代理，如下所示:

from inscriptis import get_text
import requests
url = requests.get('https://economictimes.indiatimes.com/markets/stocks/news/birla-group-enters-the-fray-to-acquire-idbi-federal-life/articleshow/64251332.cms', "lxml", headers={'User-Agent': 'Mozilla/5.0'})

print(get_text(url))

但这会出现以下错误:

AttributeError: 'Response' object has no attribute 'strip'

我该如何让这个该死的服务器停止阻止我的网络抓取？

Answer 1

您需要处理响应正文，而不是响应对象本身:

response = requests.get('https://economictimes.indiatimes.com/markets/stocks/news/birla-group-enters-the-fray-to-acquire-idbi-federal-life/articleshow/64251332.cms', "lxml", headers={'User-Agent': 'Mozilla/5.0'})

print(get_text(response.text))