python - 如何从分页中获取下一页 url。

Question

无法获取下一页的 URL。抛出回溯错误。基本上我想抓取“/browse-movies?page=2”

from bs4 import BeautifulSoup
import requests
import re
url = "https://yts.ag/browse-movies?page=1"
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")
items = soup.find_all('ul', 'tsc_pagination')[0]
for item in items:
    print item

Answer 1

您可以使用 range(1, 300) 迭代所有页面:

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent': 'Mozilla/5.0'}

for i in range(1, 300):
    url = "https://yts.ag/browse-movies?page=%s" % i

    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")
    items = soup.find_all('div', 'browse-movie-wrap')
    for item in items:
        for val in item.find_all('div','browse-movie-bottom'):
            title = item.find_all('a','browse-movie-title')[0].text
            year = item.find_all('div','browse-movie-year')[0].text
        for val in item.find_all('a','browse-movie-link'):
            try:
                rating = val.find_all('h4')[0].text
                genre = val.find_all('h4')[1].text 
            except:
                pass 

        print year, rating, genre, title

P.S. 您可能需要添加 time.sleep(1) 来减慢速度，以防他们因为过于激进地抓取其网页而阻止您的 IP。

编辑:

现在寻找下一页的URL，你可以使用正则表达式:

import re

next_page = soup.find('a', text=re.compile(r'.*Next.*'))
print next_page['href']

所以它的作用是查找内容与正则表达式'.*Next.*' 匹配的a 标记。