javascript - 使用 Selenium/BeautifulSoup 抓取动态网站

Question

我正在尝试使用 Selinium 和 Beutifulsoup 从网站上抓取评论。我试图从中抓取的网站是由 Javascript 动态生成的，这几乎超出了我在我看过的教程中学到的内容(我对 javascript 不太熟悉)。到目前为止，我最好的工作解决方案是:

browser = webdriver.Chrome(executable_path=chromedriver_path)
browser.get('https://nationen.ebcomments.dk/embed/stream?asset_id=7627366')
def load_data():
    time.sleep(1) # The site needs to load
    browser.execute_script("document.querySelector('#stream > div.talk-stream-tab-container.Stream__tabContainer___2trkn > div:nth-child(2) > div > div > div > div > div:nth-child(3) > button').click()") # Click on load more comments button

htmlSource = browser.page_source
soup = BeautifulSoup(browser.page_source, 'html.parser')
load_data() # i should call this few times to load all comments, but in this example i only do it once.
for text in soup.findAll(class_="talk-plugin-rich-text-text"):
    print(text.get_text(), "\n") # Print the comments

它有效 - 但它非常慢，我确信有更好的解决方案，特别是如果我想抓取数百篇带有评论的文章。

我认为所有的评论都是 JSON 格式的(我查看了网络下的 Chrome 开发选项卡，我可以看到有一个包含带有评论的 JSON 的响应 - 参见图片)。然后我尝试使用 SeliniumRequest 来获取数据，但完全不确定我在做什么，而且它不起作用。它说 “b'POST body missing. Did you forget to use body-parser middleware?'”。 也许我可以从评论 API 中获取 JSON，但我不确定这是否可能？

from seleniumrequests import Chrome
chromedriver_path = 'C:/chromedriver.exe'
webdriver = Chrome(executable_path=chromedriver_path)
response = webdriver.request('POST', 'https://nationen.ebcomments.dk/api/v1/graph/ql/', data={"assetId": "7627366", "assetUrl": "", "commentId": "","excludeIgnored": "false","hasComment": "false", "sortBy": "CREATED_AT", "sortOrder": "DESC"})

Answer 1

如果您只关注评论，那么以下实现应该可以帮助您:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://nationen.ebcomments.dk/embed/stream?asset_id=7627366"

with webdriver.Chrome() as driver:
    wait = WebDriverWait(driver,10)
    driver.get(link)
    while True:
        try:
            wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,".talk-load-more > button"))).click()
        except Exception: break

    for item in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"[data-slot-name='commentContent'] > .CommentContent__content___ZGv1q"))):
        print(item.text)