python - 通过使用 Beautiful Soup、Selenium 和 Pandas 提取 div 类中的值来抓取价格

Question

我试图返回给定尺寸的产品的价格，因为它们每天都会波动。我能够让我的代码在使用“类”的网站上运行，但无法使其与 div 和 span 类一起运行。

链接:https://www.flightclub.com/supreme-x-dunk-sb-low-varsity-red-varsity-red-white-black-152127?size=9.5价格:550 美元(截至本文)

from selenium import webdriver
from bs4 import BeautifulSoup                                                              
import pandas as pd                                                                        

driver = webdriver.Chrome("/Users/donlento7/chromedriver")                                 

products=[] #List to store name of the product                                             
prices=[] #List to store price of the product                                              
driver.get('https://www.flightclub.com/supreme-x-dunk-sb-low-varsity-red-varsity-red-white-black-152127?size=9.5')

content = driver.page_source                                                               
soup = BeautifulSoup(content, "lxml")                                                      
for a in soup.findAll('div',href=True, attrs={'class':'product-essential row-fluid product-type-configurable'}):
    name=a.find('div', attrs={'class':'mb-padding'})
    price=a.find('span', attrs={'class':'price'})                                      
    products.append(name.text)
    prices.append(price.text)

df = pd.DataFrame({'Product Name':products,'Price':prices})                                
#df.to_csv('products.csv', index=False, encoding='utf-8')                                  
print(df)

输出:

Empty DataFrame
Columns: [Product Name, Price]
Index: []

Answer 1

由于该行，您将获得空列表。

for a in soup.findAll('div',href=True, attrs={'class':'product-essential row-fluid product-type-configurable'}):

div 标记中没有 href 属性。

将其更改为:

for a in soup.findAll('div',attrs={'class':'product-essential row-fluid product-type-configurable'}):