python - 404页10次后如何停止Scrapy抓取？

Question

我有一个在两个数字之间抓取页面的项目。我的蜘蛛在下面。它从一个数字开始到另一个数字，然后在这些页面之间抓取。

我想让它在连续 10 个 404 页后停止。但无论如何它都必须保存 CSV 直到停止位置。

额外:是否可以让它把它停止的数字写入另一个文本文件？

这是我的示例日志:

2017-01-25 19:57:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://domain.com/entry/65848514>
{'basligi': [u'murat boz'],
 'entry': [u'<a href=https://domain.com/entry/65848514'],
 'favori': [u'0'],
 'yazari': [u'thrones']}
2017-01-25 19:57:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://domain.com/entry/65848520>
{'basligi': [u'fatih portakal'],
 'entry': [u'<a href=https://domain.com/entry/65848520'],
 'favori': [u'0'],
 'yazari': [u'agamustaf']}
2017-01-25 19:57:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://domain.com/entry/65848525> (referer: None)
2017-01-25 19:57:26 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://domain.com/entry/65848528> (referer: None)
2017-01-25 19:57:26 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://domain.com/entry/65848529> (referer: None)
2017-01-25 19:57:26 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://domain.com/entry/65848527> (referer: None)

还有我的蜘蛛:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.selector import Selector
from project.items import ProjectItem
from scrapy import Request

class MySpider(BaseSpider):
    name = "project"
    allowed_domains = ["domain.com"]
    start_urls = ["https://domain.com/entry/%d" % i for i in range(65848505,75848535)]


def parse(self, response):

    titles = HtmlXPathSelector(response).select('//li')
    for title in titles:
        item = ProjectItem()
        item['favori'] = title.select("//*[@id='entry-list']/li/@data-favorite-count").extract()
        item['entry'] = ['<a href=https://domain.com%s'%a for a in title.select("//*[@class='entry-date permalink']/@href").extract()]
        item['yazari'] = title.select("//*[@id='entry-list']/li/@data-author").extract()
        item['basligi'] = title.select("//*[@id='topic']/h1/@data-title").extract()

        return item

Answer 1

有很多方法可以做到这一点，最简单的方法可能是用回调捕获 404 错误，计算它们并在特定条件下引发 CloseSpider 异常。例如:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.selector import Selector
from project.items import ProjectItem
from scrapy import Request
from scrapy.exceptions import CloseSpider

class MySpider(BaseSpider):
    name = "project"
    allowed_domains = ["domain.com"]
    start_urls = ["https://domain.com/entry/%d" % i for i in range(65848505,75848535)]
    handle_httpstatus_list = [404] # to catch 404 with callback
    count_404 = 0


    def parse(self, response):
        if response.status == 404:
            self.count_404 += 1
            if self.count_404 == 10:
                # stop spider on condition
                raise CloseSpider('Number of 404 errors exceeded')
            return None
        else:
            self.count_404 = 0
        titles = HtmlXPathSelector(response).select('//li')
        for title in titles:
            item = ProjectItem()
            item['favori'] = title.select("//*[@id='entry-list']/li/@data-favorite-count").extract()
            item['entry'] = ['<a href=https://domain.com%s'%a for a in title.select("//*[@class='entry-date permalink']/@href").extract()]
            item['yazari'] = title.select("//*[@id='entry-list']/li/@data-author").extract()
            item['basligi'] = title.select("//*[@id='topic']/h1/@data-title").extract()

            return item

更优雅的解决方案是编写自定义下载器中间件来处理这种情况。

P.S.: Left start_urls 因为它在问题中，但是生成 10 000 000 个链接的列表并将其保存在内存中是极大的开销，您应该为 start_urls 使用任一生成器 或覆盖 start_requests。