python - 需要接受隐私政策才能访问页面

Question

我正在尝试从 AllRecipes.co.uk 抓取信息，但是当代码运行时，我没有被定向到预期的页面，而是被定向到一个要求我事先接受隐私政策的封面。这意味着我无法从我想要的页面上抓取，因为我访问的任何页面都带有此接受隐私政策封面

网站是 AllRecipes.co.uk

import requests
from bs4 import BeautifulSoup
import time
from selenium import webdriver
import numpy as np
import os


userAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
headers = {
        'user-agent': userAgent
    }

dishType = "main-recipes"
url = 'http://allrecipes.co.uk/recipes/' + dishType + '.aspx?page='
#endPage = 1259
endPage = 3
for i in range(2, endPage):
    delays = [5, 7, 9, 11, 13, 15]
    delay = np.random.choice(delays)
    time.sleep(delay)
    print("Getting request " + str(i))
    r = requests.get(url + str(i))
    soup = BeautifulSoup(r.content, "html.parser")
    print(soup)
    #names = soup.findAll('div', attrs = {'class' : "col-sm-7"})
    #for name in names:
    #    print(name)

Answer 1

您只需要设置 euConsentId cookie:

In [1]: import requests

In [2]: from bs4 import BeautifulSoup

In [3]: url = "http://allrecipes.co.uk/recipes/main-recipes.aspx?page=2"

In [4]: BeautifulSoup(requests.get(url).content, "html.parser").title.get_text()
Out[4]: 'About your privacy on this site'

In [5]: import uuid

In [6]: BeautifulSoup(requests.get(url, cookies={'euConsentId': str(uuid.uuid4())}).content, "html.parser").title.get_text()
Out[6]: 'Main course recipes - All recipes UK '

---

为了在您的代码中进行调整，我将实例化一个 "session"并在那里设置 cookie:

import uuid4

consent_id = str(uuid.uuid4())
with requests.Session() as session:
    session.cookies = {'euConsentId': consent_id}

    response = session.get(...)