python - 美丽汤中的错误 : You don't have permission to access "url" on this server.

Question

from bs4 import BeautifulSoup
import requests

site = 'https://www.france24.com/en/'

# Get HTML Content
r = requests.get(site)
soup = BeautifulSoup(r.content, 'html.parser')

print(soup)

# Find the Headline HTML
media_list = soup.find_all('div', class_='t-content__section-pb')

print(media_list)

我正在尝试从 France24 获取我正在开发的内容聚合器的 HTML。此代码适用于我尝试过的大多数网站，但由于某些原因，它会返回一些网站

You don't have permission to access "http://www.france24.com/en/" on this server.<p>
Reference #18.87e82d17.1592371984.e5f81db

Answer 1

您可以尝试在请求中传递 header 信息

site = 'https://www.france24.com/en/'
header = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36',
}
# Get HTML Content
r = requests.get(site, headers=header)