python - Selenium 检查元素是否存在并单击

Question

我正在使用 Selenium 和 Scrapy 从动态网站中抓取内容。我是 Selenium 的新手。我正在从 here 中提取酒单。该网站有一个显示更多按钮，单击该按钮会显示更多 Wine 列表。就目前而言，我只能单击按钮一次并提取酒单。但我每次都需要单击该按钮，直到不显示 show more 按钮。对此的任何帮助将不胜感激。这是到目前为止我的代码:

# -*- coding: utf-8 -*-

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from selenium import webdriver
from scrapy.selector import Selector
import time




class WineSpider(CrawlSpider):
    name = "wspider"
    allowed_domains = ["vivino.com"]



    start_urls = ["http://www.vivino.com/wineries/francis-ford-coppola/"] #hloru
    def __init__(self):
        self.driver = webdriver.Firefox()

    def parse(self,response):

        sel = Selector(self.driver.get(response.url))

        self.driver.get(response.url)
        links = []

        time.sleep(5)

        #this is for selecting the show more button

        click = self.driver.find_elements_by_xpath("//*[@id='btn-more-wines']")
        click[0].click()
        time.sleep(5)
        wines = self.driver.find_elements_by_xpath('//a[@class = "link-muted"]')
        for w in wines:
                links.append(w.get_attribute("href"))



        print len(links)
        self.driver.close()

任何帮助都会非常有用。

Answer 1

进行无限循环，使用 Explicit Wait等待“显示更多”按钮出现，一旦“显示更多”不再可见(没有更多的 Wine )就中断循环 - 然后才解析结果:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get("http://www.vivino.com/wineries/francis-ford-coppola/")

while True:
    try:
        button = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.ID, "btn-more-wines")))
    except TimeoutException:
        break  # no more wines

    button.click()  # load more wines


wines = driver.find_elements_by_xpath('//a[@class = "link-muted"]')

links = [w.get_attribute("href") for w in wines]

driver.close()

请注意，显式等待实际上是一个游戏规则改变者 - 与硬编码的 time.sleep 延迟相比，它将使您的代码更加可靠和快速。