python-3.x - 尝试使用 BeautifulSoup 抓取 amazon.in

Question

'''

    from bs4 import BeautifulSoup as soup 
    from urllib.request import urlopen as uReq

    my_url = "https://www.amazon.in/s?k=graphic+card+for+pc&crid=37UZDYFMR3TNR&sprefix=graphic+ca%2Caps%2C368&ref=nb_sb_ss_i_1_10"

    uClient = uReq(my_url)
    page_html = uClient.read()
    uClient.close()

    page_soup = soup(page_html, "html.parser")

    containers = page_soup.find_all(class_= "sg-col-20-of-24 s-result-item sg-col-0-of-12 sg-col-28-of-32 sg-col-16-of-20 sg-col sg-col-32-of-36 sg-col-12-of-16 sg-col-24-of-28")

    print(len(containers))

'''

上面的代码返回prints 0。没有选择任何东西。我究竟做错了什么？

我还尝试了以下方法:

    containers = page_soup.find_all("div",{"class": "sg-col-20-of-24 s-result-item sg-col-0-of-12 sg-col-28-of-32 sg-col-16-of-20 sg-col sg-col-32-of-36 sg-col-12-of-16 sg-col-24-of-28"})

以下是元素:

IMAGE FROM INSPECTING THE WEBPAGE

Answer 1

亚马逊不返回任何数据，如果您不指定 User-Agent在你的标题中。还有您拥有的选择器 sg-col-20-of-24 s-result-item sg-col-0-of-12 sg-col-28-of-32 sg-col-16-of-20 sg-col sg-col-32-of-36 sg-col-12-of-16 sg-col-24-of-28太长了，容易断。最好找一个较短的，例如 CSS 选择器 .s-result-item .

此代码打印标题和价格:

import requests
from textwrap import shorten
from bs4 import BeautifulSoup

url = 'https://www.amazon.in/s?k=graphic+card+for+pc&crid=37UZDYFMR3TNR&sprefix=graphic+ca%2Caps%2C368'

headers = {'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0'}
soup = BeautifulSoup(requests.get(url, headers=headers).text, 'html.parser')

print('{:<90} {}'.format('Title', 'Price'))
for result in soup.select('.s-result-item'):
    title = result.select_one('.a-text-normal').get_text(strip=True)
    price = result.select_one('.a-price-whole').get_text(strip=True)
    print('{:<90} {}'.format(shorten(title, 90), price))

打印:

Title                                                                                      Price
ASUS PH-GT1030-O2G GeForce GT 1030 2GB Phoenix Fan OC Edition HDMI DVI Graphics Card       6,249
GALAX GeForce® GTX 1050 Ti EXOC 4GB 128-bit DDR5 - DP 1.4, HDMI 2.0b, Dual Link- [...]     10,590
Gigabyte GV-RX560OC-4GD REV2.0 Radeon RX 560 OC 4GB Computer Graphics Cards                7,695
Gigabyte Geforce GT 710 2GB DDR5 Graphics Card (GV-N710D5-2GL)                             3,066
Asus Nvidia GeForce GT 710 2GB 64-Bit DDR3 PCI Express Graphic Cad with HDCP [...]         3,325
Zotac GeForce GT 1030 2GB GDDR5 64-bit Graphic card (ZT-P10300A-10L)                       6,255
ASUS GeForce GT 710 1GB GDDR5 HDMI VGA DVI Graphics Card (GT710-SL-1GD5-BRK)               2,650
Zotac GeForce GTX 1050 Ti OC Edition ZT-P10510B-10L 4GB PCI Express Graphics Card          10,994
MSI GT 710 2GD3H LP DDR3 Gaming Graphic Card                                               2,875
ASUS Cerberus GeForce GTX 1050 Ti 4GB OC Edition GDDR5 Gaming Graphics Card [...]          10,675
No Doubt Complete Set / A24FHD LED Display/Core™ i5-9400f Max Turbo Frequency 4.10 [...]   79,999
GALAX GeForce GTX 1660 Super 1-Click OC 6GB GDDR6 192-bit DP/HDMI/DVI-D Graphic Card       20,299
ZOTAC GeForce GTX 1650 OC Edition 4GB GDDR5 128-bit Gaming Graphics Card (ZT-T16500F-10L)  12,000
Zotac GT 710 2GB 64BIT DDR3 PCI-E Graphics Card                                            3,049
ASUS AREZ-PH-RX550-2G GDDR5 DP HDMI DVI AMD Graphics Cards                                 4,949
GALAX GeForce® GTX 1660 (1-Click OC) 6GB GDDR5 192-bit DP/HDMI/DVI-D Graphic Card          16,700
Zotac GT 710 2GB DDR3 Zone Edition Graphics Card                                           3,018
Gigabyte Radeon RX570 Gaming 4GB GDDR5 PCI-E, with WINDFORCE 2X with 90mm Blade Fan [...]  12,047
Gigabyte GeForce GV-N710D3-2GL 2GB PCI-Express Graphics Card (Black)                       3,087
GALAX GeForce GTX 1660 Super 1-Click OC 6GB GDDR6 192-bit DP/HDMI/DVI-D Graphic Card       20,299