python-3.x - Python/Selenium - 如何解析 URL 并单击下一页？

Question

我正在尝试解析 https://www.weforum.org/agenda/archive/covid-19 中所有文章的 href 和标题但我还想在下一页提取信息。

我的代码只能拉取当前页面，但不能处理 click() 下一页。

driver.get("https://www.weforum.org/agenda/archive/covid-19")

links =[]
titles = []

while True:
    for elem in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.tout__link'))):
        links.append(elem.get_attribute('href'))
        titles.append(elem.text)
    try:
        WebDriverWait(driver,5).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".pagination__nav-text"))).click()
        WebDriverWait(driver,5).until(EC.staleness_of(elem))
    except:
        break

谁能帮我解决这个问题？谢谢!

Answer 1

类名“pagination__nav-text”不是唯一的。根据设计，它单击第一个找到的元素，即“Prev”链接。所以你不会看到它起作用。

你能试试这个方法吗，

    driver.get("https://www.weforum.org/agenda/archive/covid-19")
wait = WebDriverWait(driver,10)

links =[]
titles = []

while True:
    for elem in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.tout__link'))):
        links.append(elem.get_attribute('href'))
        titles.append(elem.text)
    try:
        print('trying to click next')
        WebDriverWait(driver,5).until(EC.presence_of_element_located((By.XPATH,"//div[@class='pagination__nav-text' and contains(text(),'Next')]"))).click()
        WebDriverWait(driver,5).until(EC.staleness_of(elem))
    except:
        break


print(links)   
print(titles)
driver.quit()