python-如何爬过__VIEWSTATE

Question

我正在实现一个简单的 python 爬虫。我在 .aspx 站点上进行了测试，发现它没有爬过 <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKLTc2MzAxM..." />

__VIEWSTATE 的值超长。下面的每个 html 标签都没有被抓取。这是我的爬虫:

try:
    # For python 3.0 and later
    from urllib.request import Request, urlopen, URLError
except ImportError:
    # Fall back to python 2's urllib2
    from urllib2 import Request, urlopen, URLError

from HTMLParser import HTMLParser

url = "http://tickets.cathay.com.sg/index.aspx"
response = urlopen(url)
html = response.read()

# Create a subclass and override the handler methods
class MetaParser(HTMLParser):
    def handle_starttag(self, tag, attrs):
        print "Encountered a start tag:", tag
        for attr in attrs:
            print("attr:",attr)

        if tag == "img":
            for attr in attrs:
                print attr


#instantiate the parser and fed it some HTML
parser = MetaParser()
parser.feed(html)

下面是上面爬虫的结果:

Encountered a start tag: html
('attr:', ('xmlns', 'http://www.w3.org/1999/xhtml'))
Encountered a start tag: head
Encountered a start tag: title
Encountered a start tag: style
('attr:', ('type', 'text/css'))
Encountered a start tag: style
('attr:', ('type', 'text/css'))
Encountered a start tag: script
('attr:', ('language', 'javascript'))
('attr:', ('type', 'text/javascript'))
Encountered a start tag: body
Encountered a start tag: div
('attr:', ('id', 'div_loading'))
('attr:', ('style', 'display:none;'))
Encountered a start tag: b
Encountered a start tag: script
('attr:', ('language', 'javascript'))
('attr:', ('type', 'text/javascript'))
Encountered a start tag: div
('attr:', ('style', 'height:100%;width:100%;vertical-align:middle;text-align:center;'))
Encountered a start tag: br
Encountered a start tag: br
Encountered a start tag: table
('attr:', ('id', 'tbl_noJS'))
('attr:', ('cellpadding', '3'))
('attr:', ('cellspacing', '3'))
('attr:', ('class', 'asc_mb__Error'))
Encountered a start tag: tr
Encountered a start tag: th
Encountered a start tag: tr
Encountered a start tag: td
Encountered a start tag: script
('attr:', ('language', 'javascript'))
('attr:', ('type', 'text/javascript'))
Encountered a start tag: form
('attr:', ('name', 'aspnetForm'))
('attr:', ('method', 'post'))
('attr:', ('action', 'index.aspx'))
('attr:', ('id', 'aspnetForm'))
Encountered a start tag: input
('attr:', ('type', 'hidden'))
('attr:', ('name', '__VIEWSTATE'))
('attr:', ('id', '__VIEWSTATE'))
('attr:', ('value', '/wEPDwULLTExNjcwMjQ1OTIPFgIeD19fcG9zdGJhY2tjb3VudGYWAmYPZBYCAgMPZBYCAgMPZBYEZg8QZGQWAGQCAQ8QZGQWAGRk94h3o3llZzxioTZaZaEsGu8qYIM='))
Encountered a start tag: script
('attr:', ('type', 'text/javascript'))

如果您注意到，viewstate 的值与在浏览器 Page View Source 中找到的值相似但不相同。其他属性似乎也有所不同。

我在 here 找到了一个例子但它没有用。我用谷歌搜索，但找不到太多相关信息

我进一步调查并尝试抓取 http://www.microsoft.com/en-sg/default.aspx .有用!!!在查看页面源代码中，我看到它有 __VIEWSTATE。我很困惑。那为什么我的爬虫爬不到http://tickets.cathay.com.sg/index.aspx ？？

这是另一个使用 scrapy 的爬虫:

from scrapy.spider import Spider
from scrapy.selector import Selector

class MySpider(Spider):
    name = "myspider"

    start_urls = [
        "http://tickets.cathay.com.sg/index.aspx"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2]
        print "filename[", filename, "]"
        open(filename, 'wb').write(response.body)

        sel = Selector(response)

        # Using XPath query
        print sel.xpath('//img')

结果如下:

User-MacBook-Pro:tutorial User$ scrapy crawl myspider
2014-06-20 23:52:10+0800 [scrapy] INFO: Scrapy 0.22.2 started (bot: tutorial)
2014-06-20 23:52:10+0800 [scrapy] INFO: Optional features available: ssl, http11
2014-06-20 23:52:10+0800 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial'}
2014-06-20 23:52:10+0800 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-06-20 23:52:10+0800 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-06-20 23:52:10+0800 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-06-20 23:52:10+0800 [scrapy] INFO: Enabled item pipelines: 
2014-06-20 23:52:10+0800 [myspider] INFO: Spider opened
2014-06-20 23:52:10+0800 [myspider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-06-20 23:52:10+0800 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2014-06-20 23:52:10+0800 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2014-06-20 23:52:11+0800 [myspider] DEBUG: Crawled (200) <GET http://tickets.cathay.com.sg/index.aspx> (referer: None)
filename[ tickets.cathay.com.sg ]
[]
2014-06-20 23:52:11+0800 [myspider] INFO: Closing spider (finished)
2014-06-20 23:52:11+0800 [myspider] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 230,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 1856,
     'downloader/response_count': 1,
     'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2014, 6, 20, 15, 52, 11, 9068),
     'log_count/DEBUG': 3,
     'log_count/INFO': 7,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2014, 6, 20, 15, 52, 10, 960574)}
2014-06-20 23:52:11+0800 [myspider] INFO: Spider closed (finished)

使用 scrapy，我相信它使用的是 lxml，它也无法抓取 __VIEWSTATE 以下的任何内容。

Answer 1

这是一个使用 requests 和 beautifulsoup4 的工作示例(我对 scapy 的了解还不够多，无法使用它)。

import requests
from bs4 import BeautifulSoup

def get_viewstate():
    url = "http://tickets.cathay.com.sg/index.aspx"
    req = requests.get(url)
    data = req.text

    bs = BeautifulSoup(data)
    return bs.find("input", {"id": "__VIEWSTATE"}).attrs['value']

url = "http://tickets.cathay.com.sg/index.aspx"
data = {"__VIEWSTATE": get_viewstate()}
req = requests.post(url, data)

bs = BeautifulSoup(req.text)
print bs.findAll("td", {"class": "movieTitlePlatinum"}) #Just an example, you could also do bs.findAll("img") etc.