gpt4 book ai didi

python - 在 Python 中使用漂亮的汤在 html 页面上调用 "on click event"的问题

转载 作者:太空宇宙 更新时间:2023-11-03 11:32:50 25 4
gpt4 key购买 nike

我正在尝试抓取网页上所有项目的名称,但默认情况下页面上只有 18 个可见,我的代码只抓取那些。您可以通过单击“显示全部”按钮查看所有项目,但该按钮是使用 Javascript 编写的。

经过一些研究,我发现 PyQt 模块可以用来解决这个涉及 javascript 按钮的问题,我使用了它,但我仍然无法调用“点击”事件。以下是引用代码:

import csv
import urllib2
import sys
import time
from bs4 import BeautifulSoup
from PyQt4.QtGui import *
from PyQt4.QtCore import *
from PyQt4.QtWebKit import *

class Render(QWebPage):
def __init__(self, url):
self.app = QApplication(sys.argv)
QWebPage.__init__(self)
self.loadFinished.connect(self._loadFinished)
self.mainFrame().load(QUrl(url))
self.app.exec_()

def _loadFinished(self, result):
self.frame = self.mainFrame()
self.app.quit()

url = 'http://www.att.com/shop/wireless/devices/smartphones.html'
r = Render(url)
jsClick = var evObj = document.createEvent('MouseEvents');
evObj.initEvent('click', true, true );
this.dispatchEvent(evObj);


allSelector = "a#deviceShowAllLink" # This is the css selector you actually need
allButton = r.frame.documentElement().findFirst(allSelector)
allButton.evaluateJavaScript(jsClick)




page = allButton
soup = BeautifulSoup(page)
soup.prettify()
with open('Smartphones_26decv1.0.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',')
spamwriter.writerow(["Date","Day of Week","Device Name","Price"])
items = soup.findAll('a', {"class": "clickStreamSingleItem"},text=True)
prices = soup.findAll('div', {"class": "listGrid-price"})
for item, price in zip(items, prices):
textcontent = u' '.join(price.stripped_strings)
if textcontent:
spamwriter.writerow([time.strftime("%Y-%m-%d"),time.strftime("%A") ,unicode(item.string).encode('utf8').strip(),textcontent])

我遇到的错误如下:

"Invalid Syntax" Error for evObj

谁能帮我调用这个“onclick”事件,这样我就可以抓取所有项目的数据。请原谅我的无知,因为我是编程新手。

最佳答案

from contextlib import closing
from selenium.webdriver import Firefox # pip install selenium
from selenium.webdriver.support.ui import WebDriverWait
from BeautifulSoup import BeautifulSoup
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# use firefox to get page with javascript generated content
with closing(Firefox()) as driver:
driver.get("http://www.att.com/shop/wireless/devices/smartphones.html")
button = driver.find_element_by_id('deviceShowAllLink')
button.click()
# wait for the page to load
element = WebDriverWait(driver, 10).until(
EC.invisibility_of_element_located((By.ID, "deviceShowAllLink"))
)
# store it to string variable
page_source = driver.page_source

soup = BeautifulSoup(page_source)
items = soup.findAll('div', {"class": "list-item"})
print "items count:",len(items)

这会有帮助吗?

关于python - 在 Python 中使用漂亮的汤在 html 页面上调用 "on click event"的问题,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/13967454/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com