selenium - Selenium 没有这种元素异常

Question

我正在尝试使用以下方式刮除Glassdoor的评论:
https://github.com/MatthewChatham/glassdoor-review-scraper

但是我得到了错误并且不知道如何克服它。请帮忙。

在Windows 10和Chrome 77上

2019-09-11 08:48:58,961信息377:main.py(1824)-配置浏览器

DevTools listening on ws://127.0.0.1:#####/devtools/browser/cccb51a6-3dc2-4f06-90db-################

2019-09-11 08:49:03,942 INFO 419    :main.py(1824) - Scraping up to 1000 reviews.
2019-09-11 08:49:03,946 INFO 358    :main.py(1824) - Signing in to email@gmail.com
2019-09-11 08:49:06,541 INFO 339    :main.py(1824) - Navigating to company reviews
2019-09-11 08:49:12,674 INFO 286    :main.py(1824) - Extracting reviews from page 1
2019-09-11 08:49:12,696 INFO 291    :main.py(1824) - Found 10 reviews on page 1
2019-09-11 08:49:12,840 WARNING 126    :main.py(1824) - Failed to scrape employee_title
Traceback (most recent call last):
  File "main.py", line 461, in <module>
    main()
  File "main.py", line 441, in main
    reviews_df = extract_from_page()
  File "main.py", line 295, in extract_from_page
    data = extract_review(review)
  File "main.py", line 281, in extract_review
    res[field] = scrape(field, review, author)
  File "main.py", line 264, in scrape
    return fdict[field](review)
  File "main.py", line 156, in scrape_years
    'reviewBodyCell').find_element_by_tag_name('p')
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 305, in find_element_by_tag_name
    return self.find_element(by=By.TAG_NAME, value=name)
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 659, in find_element
    {"using": by, "value": value})['value']
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Users\dvnguyen\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"p"}
  (Session info: headless chrome=77.0.3865.75)

Answer 1

glassdoor可能已更改其网站/布局，并且此刮板使用的方法不再有效地检测页面。

就是说，它正在类类型reviewBodyCell的某个元素内寻找简单的“p”标签。

File "main.py", line 156, in scrape_years
    'reviewBodyCell').find_element_by_tag_name('p')

快速浏览一下该网站，看起来在某些情况下，类名为reviewBodyCell的元素下没有p标记。