python - 使用 Beautfiul Soup 提取当前出价金额 - 动态更新值

Question

这是我对 Python 和 BeautifulSoup 的第一次介绍。我试图从一个流行的拍卖网站 (RealInsight) 上列出的特定属性中获取当前的出价金额，但我无法让 BeautifulSoup 提取我正在寻找的实际整数，只有 HTML 代码。我正在寻找“s-b-n”类标签的值(value)，在拍卖真正开始之前，它的值(value)是 3,250,000 美元。

https://marketplace.realinsight.com/sales/details/XXX

我认为这是因为该值是动态更新的并且是在 HTML 代码之外生成的，但我不确定如何验证该论点或如果证明正确则如何获取该值。我还认为我可能没有正确引用包含值的表，但同样，我在 python 或 bs4 方面经验不足。

[使用 ewwink 的方法使用 ewwink 的方法对下面的最终代码进行了更新 - 一次抓取一次，持续五秒] - 已更新以处理拍卖结束 -

import bs4
import time
import csv
import datetime
import sys
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq
my_url = 'https://marketplace.realinsight.com/sales/details/XXX'
uclient = uReq(my_url)
page_html = uclient.read()
uclient.close()
endmsg = "Auction End"
page_soup = soup(page_html, "html.parser")
propname = page_soup.title.text
bids = page_soup.select_one(".body-content")
currentbid = bids['data-nb']
bidincrement = bids['data-bi']
with open(propname + '_bids.csv','w',newline='') as f:
    thewriter = csv.writer(f)
    thewriter.writerow(['i','prop_name','date_time','bid_increment','bid_amt'])
    for i in range(0,5,1):
            try:
                import sys
                sale = page_soup.select_one("div.sale-end-text")
                auctionend = sale.text.replace(" ", "")
                if auctionend == sale.text.replace(" ", ""):
                    currentDT = datetime.datetime.now()    
                    thewriter.writerow([i,endmsg,currentDT,currentbid])    
                    print(endmsg,currentbid)
                    time.sleep(1)
                    sys.exit()
                else:
                    print('will never get to this point')
            except Exception:
                pass

            currentDT = datetime.datetime.now()
            thewriter.writerow([i,propname,currentDT,bidincrement,currentbid])
            print(i,propname,currentDT,bidincrement,currentbid)
            time.sleep(1)

更新了 chitown88 的方法

import bs4
import datetime
import time
import csv
import selenium
from selenium import webdriver
driver = webdriver.Chrome(executable_path='C:\\Users\\XXXX\\Downloads\\chromedriver_win32\\chromedriver.exe')
driver.get('https://marketplace.realinsight.com/sales/details/XXX')
html = driver.page_source
page_soup = bs4.BeautifulSoup(html,"html.parser")
bids = page_soup.select("td.s-b-n")
propname = page_soup.title.text
currentbid = bids[0].text
with open(propname + '_bids.csv','w',newline='') as f:
    thewriter = csv.writer(f)
    thewriter.writerow(['i','prop_name','date_time','bid_amt'])
    for i in range(0, 5, 1):
        currentDT = datetime.datetime.now()
        driver.refresh()
        thewriter.writerow([i, propname, currentDT, currentbid])
        print(i, propname, currentDT, currentbid)
        time.sleep(1)
driver.close()

我可以在 HTML 代码中看到我要查找的数字 ($3,250,000)，但它每隔几秒就会闪烁和更新一次，这就是为什么我认为它是在其他地方生成的。

任何指导将不胜感激。

Answer 1

您需要在解析之前加载页面。 Selenium非常适合。

import bs4 
from selenium import webdriver 

driver = webdriver.Chrome()
driver.get('https://marketplace.realinsight.com/sales/details/367')

html = driver.page_source
page_soup = bs4.BeautifulSoup(html,"html.parser")

bids = page_soup.select("td.s-b-n")
bid = bids[0].text
print(bid)

driver.close()

和输出:

In [91]: print(bid)
$3,250,000