python - 尽管继承自 CrawlSpider，但 Scrapy NotImplemented 解析错误

Question

我正在尝试让一个非常简单的 Scrapy 爬虫工作。我从解析中收到 NotImplemented 错误 - 本质上与此相同的问题:Why does scrapy throw an error for me when trying to spider and parse a site?

除了我继承自CrawlSpider。

我或多或少地逐字关注了这个页面: https://github.com/scrapy/scrapy/blob/master/docs/topics/spiders.rst#crawlspider-example

要编写此代码:

import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor


class SiteSpider(CrawlSpider):

    name = 'sdreader'
    allowed_domains = ['sandiegoreader.com']
    start_urls = ['http://www.sandiegoreader.com/events/all/']
    rules = [Rule(LinkExtractor(allow=['/events/2015/.+', '/events/ongoing/.+']), 'parse_event')]

    def parse_event(self, response):
        event = EventItem()
        event['name'] = response.xpath('//*[@id="content"]/div[2]/h2/text()').extract()
        return event

我在日志中得到了这个:

2015-04-12 15:01:07-0700 [scrapy] INFO: Scrapy 0.24.5 started (bot: sdreader)
2015-04-12 15:01:07-0700 [scrapy] INFO: Optional features available: ssl, http11
2015-04-12 15:01:07-0700 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'sdreader.spiders', 'SPIDER_MODULES': ['sdreader.spiders'], 'DEPTH_LIMIT': 1, 'DOWNLOAD_DELAY': 2, 'BOT_NAME': 'sdreader'}
2015-04-12 15:01:07-0700 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-04-12 15:01:07-0700 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-04-12 15:01:07-0700 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, DeltaFetch, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-04-12 15:01:07-0700 [scrapy] INFO: Enabled item pipelines:
2015-04-12 15:01:07-0700 [sdreader] INFO: Spider opened
2015-04-12 15:01:07-0700 [sdreader] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-04-12 15:01:07-0700 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-04-12 15:01:07-0700 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2015-04-12 15:01:24-0700 [sdreader] DEBUG: Crawled (200) <GET http://www.sandiegoreader.com/events/all/> (referer: None)
2015-04-12 15:01:24-0700 [sdreader] ERROR: Spider error processing <GET http://www.sandiegoreader.com/events/all/>
    Traceback (most recent call last):
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/base.py", line 1178, in mainLoop
        self.runUntilCurrent()
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/base.py", line 800, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 368, in callback
        self._startRunCallbacks(result)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 464, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 551, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/Library/Python/2.7/site-packages/scrapy/spider.py", line 56, in parse
        raise NotImplementedError
    exceptions.NotImplementedError:

2015-04-12 15:01:24-0700 [sdreader] INFO: Closing spider (finished)
2015-04-12 15:01:24-0700 [sdreader] ERROR: Error caught on signal handler: <bound method ?.spider_closed of <sdreader.middlewares.deltafetch.DeltaFetch object at 0x105167410>>
    Traceback (most recent call last):
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 551, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 275, in <lambda>
        spider=spider, reason=reason, spider_stats=self.crawler.stats.get_stats()))
      File "/Library/Python/2.7/site-packages/scrapy/signalmanager.py", line 23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/Library/Python/2.7/site-packages/scrapy/utils/signal.py", line 53, in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 134, in maybeDeferred
        result = f(*args, **kw)
      File "/Library/Python/2.7/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 54, in robustApply
        return receiver(*arguments, **named)
      File "/Users/jasonyau/events/weekendr-db/sdreader/scrapy/sdreader/sdreader/middlewares/deltafetch.py", line 65, in spider_closed
        self.db.close()
      File "/Library/Python/2.7/site-packages/pymongo/collection.py", line 1705, in __call__
        self.__name)
    exceptions.TypeError: 'Collection' object is not callable. If you meant to call the 'close' method on a 'Database' object it is failing because no such method exists.

2015-04-12 15:01:24-0700 [sdreader] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 232,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 114076,
     'downloader/response_count': 1,
     'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2015, 4, 12, 22, 1, 24, 154684),
     'log_count/DEBUG': 3,
     'log_count/ERROR': 2,
     'log_count/INFO': 7,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'spider_exceptions/NotImplementedError': 1,
     'start_time': datetime.datetime(2015, 4, 12, 22, 1, 7, 718810)}
2015-04-12 15:01:24-0700 [sdreader] INFO: Spider closed (finished)

一定是某处的用户错误。我研究了 Crawlspider 源代码，它似乎达到了我的预期。它正在实现解析并继承自Spider。

Answer 1

尝试将规则中的回调函数更改为:

rules = [Rule(LinkExtractor(allow=['/events/2015/.+', '/events/ongoing/.+']), self.parse_event)]