python - 在 AJAX 请求完成后抓取页面

Question

我正在抓取:

http://www.wotif.com/hotel/View?hotel=W3830&page=1&adults=2&startDay=2014-11-08&region=1&descriptionSearch=true#property-reviews

使用以下代码:

hotel_page  = requests.get(hotel_url).text
hotel_page_soup = BeautifulSoup(hotel_page)

但是，这不包括 Guest Review部分，原因是它是在页面加载后通过 AJAX 调用加载的。

问题:如何在所有 AJAX 调用完成后才抓取页面？

Answer 1

你需要调用这个 URL以及确保 X-Requested-With 是 XMLHttpRequest

URL="http://www.wotif.com/review/fragment?propertyId=W3830&limit=5"

headers={"X-Requested-With":"XMLHttpRequest",
"User-Agent":"Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"}

r=requests.get(URL,headers=headers)

#response here will be in json format
#Page source can be extracted using key `html'`
response=r.json()['html']
soup=BeautifulSoup(response)
reviews=soup.find(class_="review-score review-score-large").text
print reviews

Out[]:u'\n\n4.4\nOut of 5\n\n\n'

print reviews.strip()

Out[]:u'4.4\nOut of 5'