python-3.x - 如何从通过 Javascript 加载的页面上抓取数据

Question

我想使用beautifulsoup - https://www.x....s.com/video_id/the-suburl 从这个页面上刮掉评论
评论通过 Javascript 点击加载。评论是分页的，每个页面也会在点击时加载评论。我希望获取所有评论，对于每条评论，我想获取海报个人资料网址，评论，否。喜欢、不喜欢和发布时间(如页面上所述)。
注释可以是字典列表。
我该怎么做？

Answer 1

此脚本将打印在页面上找到的所有评论:

import json
import requests
from bs4 import BeautifulSoup


url = 'https://www.x......com/video_id/gggjggjj/'
video_id = url.rsplit('/', maxsplit=2)[-2].replace('video', '')

u = 'https://www.x......com/threads/video/ggggjggl/{video_id}/0/0'.format(video_id=video_id)
comments = requests.post(u, data={'load_all':1}).json()

for id_ in comments['posts']['ids']:
    print(comments['posts']['posts'][id_]['date'])
    print(comments['posts']['posts'][id_]['name'])
    print(comments['posts']['posts'][id_]['url'])
    print(BeautifulSoup(comments['posts']['posts'][id_]['message'], 'html.parser').get_text())
    # ...etc.
    print('-'*80)