python - BaseSpider 和 CrawlSpider 的区别

Question

我一直在努力理解在网络抓取中使用 BaseSpider 和 CrawlSpider 的概念。我读过 docs.但是BaseSpider上没有提及。如果有人能解释一下 BaseSpider 和 CrawlSpider 之间的区别，那将对我很有帮助。

Answer 1

BaseSpider 是以前存在的东西，现在已弃用(自 0.22 起)- 使用 scrapy.Spider 代替:

import scrapy

class MySpider(scrapy.Spider):
    # ...

scrapy.Spider是最简单的蜘蛛，它基本上会访问 start_urls 中定义的 URL 或 start_requests() 返回的 URL。

使用CrawlSpider当您需要“爬行”行为时 - 提取链接并关注它们:

This is the most commonly used spider for crawling regular websites, as it provides a convenient mechanism for following links by defining a set of rules. It may not be the best suited for your particular web sites or project, but it’s generic enough for several cases, so you can start from it and override it as needed for more custom functionality, or just implement your own spider.