python - Scrapy 显示 notImplementedError 我不知道为什么

Question

我的 Scrapy 代码不起作用，我不确定为什么。我的蜘蛛是在 Reddit 上抓取权力的游戏 subreddit 的测试。

这是我的代码:

import scrapy


class Redditbot2Spider(scrapy.Spider):
    name = 'redditbot2'
    allowed_domains = ['www.reddit.com']
    start_urls = ['https://www.reddit.com/r/gameofthrones/']


def parse(self, response):
    titles = response.selector.xpath('//h2/text()').extract()
    votes = response.selector.xpath('//div[@class="_1rZYMD_4xY3gRcSS3p80D0"]/test()').extract()
    time = response.selector.xpath('//a[@class="_3jOxDPIQ0KaOWpzvSQo-1s"]/text()').extract()
    comments = response.selector.xpath('//span[@class="FHCV02u6Cp2zYL0fhQPsO"])/text()').extract()

    for item in zip(titles, votes, time, comments):
        scraped_info = {
            'title': titles[0],
            'vote': votes[1],
            'time': time[2],
            'comments': comments[3],
        }
        yield scraped_info

这里是错误日志:

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py", line 90, in parse
    raise NotImplementedError('{}.parse callback is not defined'.format(self.__class__.__name__))
NotImplementedError: Redditbot2Spider.parse callback is not defined

Redditbot2Spider.parse 方法在那里，所以我不明白它为什么这么说。有什么想法吗？

Answer 1

您的代码存在缩进问题，parse 方法与类处于同一级别，因此解释器没有意识到它是类的成员。您必须在 parse 方法中缩进:

class Redditbot2Spider(scrapy.Spider): 
    name = 'redditbot2'
    allowed_domains = ['www.reddit.com']
    start_urls = ['https://www.reddit.com/r/gameofthrones/']
    def parse(self, response): 
        titles = response.selector.xpath('//h2/text()').extract()
        #etc.