python - 我如何在 Python 或 Ruby 上查询没有 Selenium 的结果

Question

我正在尝试跟踪门票趋势

目前，我正在使用 selenium 来模拟提交表单。

如您所知，selenium 速度慢且消耗更多内存。

但是，当您提交表单时，它会将您重定向到一个新的 url http://makeabooking.flyscoot.com/Flight/Select

因此，我不知道如果没有 Selenium 我怎么能做到这一点。

因为我无法像这样更改查询形式 http://makeabooking.flyscoot.com/Flight/from={TPE}&to={NYK}&date={2015-10-12} 获取结果。

有没有想过使用支持 SSL 代理和 HTTP 代理的 Ruby 或 Python 来做到这一点？

示例网站:http://www.flyscoot.com/index.php/en/

Answer 1

您可以轻松地从 chrome 获取 curl 请求并通过以下方式使用它:

F12 > Network > request > Right Click > Copy As cURL 

curl 'http://makeabooking.flyscoot.com/Flight/Select' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,tr;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8' -H 'Referer: http://www.flyscoot.com/index.php/en/' -H 'Cookie: optimizelyEndUserId=oeu1444666692081r0.12463579000905156; __utmt=1; granify.lasts@1345=1444666699786; ASP.NET_SessionId=lql5yzv1l3yatkh1lcumg2e5; dotrez=1209262602.20480.0000; optimizelySegments=%7B%222335550040%22%3A%22gc%22%2C%222344180004%22%3A%22referral%22%2C%222354350067%22%3A%22false%22%2C%222355380121%22%3A%22none%22%7D; optimizelyBuckets=%7B%223025070068%22%3A%223020800213%22%7D; __utma=185425846.733949751.1444666694.1444666694.1444666694.1; __utmb=185425846.2.10.1444666694; __utmc=185425846; __utmz=185425846.1444666694.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/33084039/how-could-i-query-the-result-without-selenium-on-python-or-ruby; granify.uuid=68b0d8e8-d068-40d8-9068-3098e870b858; granify.session@1345=1444666699786; granify.flags@1345=8; _gr_ep_sent=1; _gr_er_sent=1; granify.session_init@1345=2; optimizelyPendingLogEvents=%5B%5D' -H 'Connection: keep-alive' -H 'X-FirePHP-Version: 0.0.6' -H 'Cache-Control: max-age=0' --compressed

如果您可以正确设置headers 和cookies 信息，您就可以使用Python 请求。如果要将其转换为 Python 请求，可以使用 this link .通过这种方式，您可以模拟浏览器。查看 pyton 请求:

cookies = {
    'optimizelyEndUserId': 'oeu1444666692081r0.12463579000905156',
    '__utmt': '1',
    'granify.lasts@1345': '1444666699786',
    'ASP.NET_SessionId': 'lql5yzv1l3yatkh1lcumg2e5',
    'dotrez': '1209262602.20480.0000',
    'optimizelySegments': '%7B%222335550040%22%3A%22gc%22%2C%222344180004%22%3A%22referral%22%2C%222354350067%22%3A%22false%22%2C%222355380121%22%3A%22none%22%7D',
    'optimizelyBuckets': '%7B%223025070068%22%3A%223020800213%22%7D',
    '__utma': '185425846.733949751.1444666694.1444666694.1444666694.1',
    '__utmb': '185425846.2.10.1444666694',
    '__utmc': '185425846',
    '__utmz': '185425846.1444666694.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/33084039/how-could-i-query-the-result-without-selenium-on-python-or-ruby',
    'granify.uuid': '68b0d8e8-d068-40d8-9068-3098e870b858',
    'granify.session@1345': '1444666699786',
    'granify.flags@1345': '8',
    '_gr_ep_sent': '1',
    '_gr_er_sent': '1',
    'granify.session_init@1345': '2',
    'optimizelyPendingLogEvents': '%5B%5D',
}

headers = {
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'en-US,en;q=0.8,tr;q=0.6',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Referer': 'http://www.flyscoot.com/index.php/en/',
    'Connection': 'keep-alive',
    'X-FirePHP-Version': '0.0.6',
    'Cache-Control': 'max-age=0',
}

requests.get('http://makeabooking.flyscoot.com/Flight/Select', headers=headers, cookies=cookies)

如果保存结果，可以通过浏览器看到结果(打开 stack.html):

r = requests.get('http://makeabooking.flyscoot.com/Flight/Select', headers=headers, cookies=cookies
f = open("stack1.html", "w")
f.write(r.content)