gpt4 book ai didi

python-3.x - python : Access Denied at Random Points When Using Requests

转载 作者:行者123 更新时间:2023-12-03 16:42:36 25 4
gpt4 key购买 nike

我正在使用 requests 和 beautifulsoup 浏览流行的漫画商店 comixology,以便列出所有漫画的标题和问题以及发布日期,因此我请求了大量的网页。不幸的是,中途我会得到错误:

您无权访问此服务器上的(URL)

我尝试使用递归尝试请求的函数。但这不起作用我没有把整个代码放进去,因为它很长。

def getUrl(url):
try:
page = requests.get(url)
except:
getUrl(url)
return page

最佳答案

The User-Agent request header contains a characteristic string that allows the network protocol peers to identify the application type, operating system, software vendor or software version of the requesting software user agent. Validating User-Agent header on server side is a common operation so be sure to use valid browser’s User-Agent string to avoid getting blocked.

(来源:http://go-colly.org/articles/scraping_related_http_headers/)

您唯一需要做的就是设置一个合法的用户代理。因此,添加 header 以模拟浏览器。 :

# This is a standard user-agent of Chrome browser running on Windows 10 
headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36' }

例子:

from bs4 import BeautifulSoup
import requests
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
resp = requests.get('http://example.com', headers=headers).text
soup = BeautifulSoup(resp, 'html.parser')

此外,您可以添加另一组 header 来伪装成合法浏览器。像这样添加更多标题:

headers = { 
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language' : 'en-US,en;q=0.5',
'Accept-Encoding' : 'gzip',
'DNT' : '1', # Do Not Track Request Header
'Connection' : 'close'
}

关于python-3.x - python : Access Denied at Random Points When Using Requests,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/54373305/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com