python - 使用 selenium 执行事件和抓取

Question

我对 selenium 还很陌生，我期待执行一个扩展表格的 click() 按钮，之后我期待着抓取扩展表格的内容。

这是我尝试过的:

url = 'https://www.telegraph.co.uk/markets-hub/assets/shares/'
phantomjs_path = '/usr/local/bin/phantomjs'
driver = webdriver.PhantomJS(executable_path=phantomjs_path,service_args=['--load-images=no'])
driver.get(url)

element = driver.find_element_by_xpath("//div[@class='tmg-show-more']")

html_source = driver.page_source
element.click()
driver.quit()
soup = BeautifulSoup(html_source,"html5lib")
tables=soup.find('table',class_="table-static kurser-table")
link=pd.read_html(str(tables),flavor='html5lib',thousands='.',decimal=',',header=0)



print(link)

当前输出如下:

raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: {"errorMessage":"Element is not currently visible and may not be manipulated",
"request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"81","Content-
Type":"application/json;charset=UTF-8","Host":"127.0.0.1:58447","User-Agent":"Python http auth"},
"httpVersion":"1.1","method":"POST","post":"{\"id\": \":wdc:1556893378248\", \"sessionId\": \"f223cb80-6dae-11e9-b00b-4bb158952171\"}",
"url":"/click","urlParsed":{"anchor":"","query":"","file":"click","directory":"/","path":"/click","relative":"/click","port":"","host":"","password":"","user":"",
"userInfo":"","authority":"","protocol":"","source":"/click","queryKey":{},"chunks":["click"]},"urlOriginal":"/session/f223cb80-6dae-11e9-b00b-4bb158952171/element/:wdc:1556893378248/click"}}
Screenshot: available via screen

Answer 1

您需要向下滚动以使 selenium Web 驱动程序知道该元素实际存在的位置。

代码:

driver.get("https://www.telegraph.co.uk/markets-hub/assets/shares/")

wait = WebDriverWait(driver, 10)

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.section-box.kurser-table-container')))

element = driver.find_element_by_xpath("//a[@ng-click='stocksShowMore()']")

actions = ActionChains(driver)
actions.move_to_element(element).perform()

element.click()

进口:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.action_chains import ActionChains