python - 努力使用 Selenium 刮 table

Question

所以我期待着对这个 link 中出现的表格进行抓取.

为了抓取，我决定使用 Selenium 。

在我的第一次尝试中，我所做的是:

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)
html_source = self.driver.page_source
self.driver.quit()
BeautifulSoup(html_source, "html5lib")
table = soup.find('table', {'class': 'heavy-table ncpulse-fav-table ncpulse-sortable compressed-table'})
df = pd.read_html(str(table), flavor='html5lib', header=0, thousands='.', decimal=',')

但是它输出错误

'no tables found'

然后我尝试使用 expected_conditions 类，因为当我在 SO 中查找时可能是“页面源甚至在子元素完全呈现之前就被拉出”

因此我尝试了这样的事情:

driver.get(route)
element_present = expected_conditions.presence_of_element_located(
    (By.CLASS_NAME, 'heavy-table ncpulse-fav-table ncpulse-sortable compressed-table'))
WebDriverWait(driver, 20).until(element_present)
html_source = driver.page_source 
driver.quit()

然而这次它输出:

selenium.common.exceptions.TimeoutException: Message

因此我的问题是:如何获得所需的输出？我在使用 expected_conditions 类时做错了什么？到底是什么问题/前端技术使得它如此艰难地抢占席位？

Answer 1

从表中提取内容作为<table>是Angular基于元素使用 Selenium和 python而不是 presence_of_element_located()你必须诱导WebDriverWait对于 visibility_of_element_located()您可以使用以下任一项 Locator Strategies :

• 使用 CSS_SELECTOR :

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.heavy-table.ncpulse-fav-table.ncpulse-sortable.compressed-table"))).text)
  

• 使用 XPATH :

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='heavy-table ncpulse-fav-table ncpulse-sortable compressed-table']"))).text)
  

• 注意:您必须添加以下导入:

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• 控制台输出:

  AKTIE +/- +/-% SENESTE ÅTD% BUD UDBUD VOLUMEN OMSÆTNING MARKEDSVÆRDI TID
  Abn Amro Bank N.V. -0,32 -4,08% 7,48 -53,90% - - 7,9 mio 59,0 mio - 21:09
  Adyen 81,00 5,62% 1523,00 108% - - 954 082 1,5 mia - 21:09
  Aegon -0,08 -3,49% 2,16 -45,47% - - 17,4 mio 37,5 mio - 21:05
  Ahold Del 0,25 0,98% 25,65 19,74% - - 8,0 mio 204,1 mio - 21:05
  Akzo Nobel 0,14 0,16% 85,86 -3,16% - - 1,1 mio 90,6 mio - 21:06
  Arcelormittal Sa 0,08 0,66% 11,53 -26,26% - - 11,9 mio 137,3 mio - 21:09
  Asm International 0,35 0,29% 119,10 21,23% - - 403 117 48,0 mio - 21:07
  Asml Holding 1,50 0,49% 308,45 17,56% - - 2,3 mio 712,7 mio - 21:05
  Asr Nederland -0,22 -0,73% 29,76 -4,97% - - 740 781 22,0 mio - 21:05
  Dsm Kon 2,25 1,66% 138,20 21,52% - - 680 867 94,1 mio - 21:09
  Galapagos -1,45 -1,22% 117,70 -36,89% - - 475 793 56,0 mio - 21:05
  Heineken 0,74 0,94% 79,10 -15,50% - - 1,1 mio 88,0 mio - 21:05
  Imcd 1,85 1,80% 104,85 36,23% - - 922 391 96,7 mio - 21:05
  Ing Groep N.V. -0,19 -2,80% 6,60 -38,24% - - 43,4 mio 286,2 mio - 21:08
  Just Eat Takeaway 0,08 0,09% 91,70 11,56% - - 1,1 mio 100,2 mio - 21:09
  Kpn Kon -0,03 -1,54% 2,11 -15,04% - - 21,4 mio 45,1 mio - 21:05
  Nn Group -0,35 -1,06% 32,80 3,82% - - 2,4 mio 79,6 mio - 21:05
  Philips Kon -0,08 -0,20% 39,42 -9,42% - - 5,2 mio 205,9 mio - 21:05
  Prosus -1,52 -1,89% 78,74 18,35% - - 15,0 mio 1,2 mia - 21:09
  Randstad Nv -0,98 -2,09% 45,93 -15,63% - - 698 496 32,1 mio - 21:05
  Relx 0,00 0,03% 19,64 -10,24% - - 1,9 mio 36,6 mio - 21:06
  Royal Dutch Shella -0,24 -2,07% 11,45 -54,58% - - 21,1 mio 241,2 mio - 21:07
  Unibail-Rodamco-We -3,79 -10,53% 32,20 -75,02% - - 6,6 mio 213,2 mio - 21:07
  Unilever -1,04 -2,00% 50,98 2,00% - - 8,2 mio 417,5 mio - 21:08
  Wolters Kluwer -0,04 -0,05% 72,88 14,19% - - 803 644 58,6 mio - 21:05