javascript - JS 网页抓取找不到某些元素

Question

我想使用 Dryscrape 从 Google 抓取酒店客房价格结果。例如这里的rhs_block https://www.google.co.uk/search?q=The+Taj+Mahal+Palace+hotel

但是它似乎没有渲染然后收集 JavaScript，我想知道我可能会在哪里出错。？

import dryscrape
from bs4 import BeautifulSoup

dryscrape.start_xvfb()
session = dryscrape.Session()

my_url = 'https://www.google.ie/search?q=The+Taj+Mahal+Palace+hotel'
session.visit(my_url)
response = session.body()
soup = BeautifulSoup(response, "lxml")

# prices = soup.find('div', {"class" : "rhs_block"})
prices = soup.find('div', {"class" : "lhpr-content-item"})

print prices

我已经在一个简单的 js 渲染页面上对此进行了测试，所以它确实有效。任何指点将不胜感激，因为 dryscrape 对我来说相当新。

Answer 1

Google 不喜欢您的用户代理设置。在这一行下:

session = dryscrape.Session()

添加行:

session.set_header("User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:41.0) Gecko/20100101 Firefox/41.0")

然后会输出:

<div class="lhpr-content-item" data-key="8"><div class="_qS"><a class="_dkf" data-dp="€176" data-pid="8" data-ved="0ahUKEwisoqOeisjVAhXCI1AKHWhQAm0QwDEIzAEoBDAW" href="https://www.google.com/travel/clk?pc=AA80Osxnd1Ycj04hDym-ZpFIn9a-iLsqE7UNxtLtnVS5khTT2PvlxyLaBSJZKt9V3zLJWmUBQJedYFG2CzsGB4Ru572oiGIF3i-UYsg1BBFNbDFPhXelW-FNo6lefLaSbCcPqO1W6rOEQT_ev6stedzfqyjT2Y7QnMNz5TGkr1zDWIfI6iQgV2l7mcMhzxHV7GKVjTjhX6KL-CT3c_9wBPpKVa1MICyikHUOf72incZ6e9TF1aMGcNKf6W91fdU__ZJOv3jByF7bkPQNOWM" onmousedown="return rwt(this,'','','','23','AFQjCNG0CN8A7n-gxtETpYwsGydozaH1Yg','','0ahUKEwisoqOeisjVAhXCI1AKHWhQAm0QwDEIzAEoBDAW','','',event)"><img alt="The Taj Mahal Palace" class="_Tjf" data-deferred="1" id="zemJWeyHEsLHwALooInoBguid_8" onload="google&amp;&amp;google.aft&amp;&amp;google.aft(this)" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAAAXNSR0IArs4c6QAAAS9JREFUOBFjTJ9v7sXwn2EmIyOjDAMF4P///08YGBnSmahhGMgdYAcBHcZEqcuQPQUyiwlZgBpsuIGTYw8xTE84zsDMxEKRuXADKTIFSTNW57AwsTIk2zcxaEiZMVx+fJRhweFGhn///4J9gKQXzPz84z3DxrMzGI7e3gTmY3WhubIng4G8AwMHKxeDqZIrg7GCM7o5cD4vhyBDsGkenI/VQJBrkAE6H1kOxAZZDANYvXzy7g4GDUkTiJefHGU493A/TD1BGquBIBfNB4YbOQDDwClxh8kxB64Hw8C///7AJYlhoKdbDAPzlzgykGIoKDMgA6yxjKyAVDbVDWTMmG/xn1RXIKtnYmQGloBA+P8fWBgjDJEVE8NGT/RM4JKWGJ1EqAGZxQQqtqlhKKwKAAB1/VzfnOVWVwAAAABJRU5ErkJggg=="/><div class="_uFf"><img alt="book action chevron" class="_hEj" onload="google&amp;&amp;google.aft&amp;&amp;google.aft(this)" src="https://www.gstatic.com/images/icons/material/system/2x/chevron_right_grey600_24dp.png"/><div class="_akf"><span><span class="_bkf"><span class="_FQr"><span class="_V0p">€176</span></span></span></span></div><div class="_zbu"><span class="_Zjf">The Taj Mahal Palace</span><span class="_aMr"><span class="_bMr"> · </span><span>Official website</span></span></div></div></a></div></div>