python - 如何从无限滚动网页中抓取正确数量的 URL？

Question

我正在尝试从网页中抓取 URL。我正在使用这段代码:

from bs4 import BeautifulSoup

import urllib2 

url = urllib2.urlopen("http://www.barneys.com/barneys-new-york/men/clothing/shirts/dress/classic#sz=176&pageviewchange=true")

content = url.read()
soup = BeautifulSoup(content)

links=soup.find_all("a", {"class": "thumb-link"})

for link in links:

      print (link.get('href'))

但我得到的输出只有 48 个链接，而不是 176 个。我做错了什么？

Answer 1

所以我所做的是使用 Postmans 拦截器功能来查看网站每次加载下一组 36 件衬衫时发出的调用。然后从那里复制代码中的调用。你不能一次转储所有 176 个项目，所以我一次复制了 36 个网站。

from bs4 import BeautifulSoup
import requests

urls = []

for i in range(1, 5):
    offset = 36 * i
    r = requests.get('http://www.barneys.com/barneys-new-york/men/clothing/shirts/dress/classic?start=1&format=page-element&sz={}&_=1434647715868'.format(offset))
    soup = BeautifulSoup(r.text)

    links = soup.find_all("a", {"class": "thumb-link"})

    for link in links:
        if len(urls) < 176:
            print (link.get('href'))
            urls.append(link.get('href'))