scrapy - 使用 Scrapy 抓取多个 URL

Question

如何使用 Scrapy 抓取多个 URL？

我是否被迫制作多个爬虫？

class TravelSpider(BaseSpider):
    name = "speedy"
    allowed_domains = ["example.com"]
    start_urls = ["http://example.com/category/top/page-%d/" % i for i in xrange(4),"http://example.com/superurl/top/page-%d/" % i for i in xrange(55)]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        items = []
        item = TravelItem()
        item['url'] = hxs.select('//a[@class="out"]/@href').extract()
        out = "\n".join(str(e) for e in item['url']);
        print out

Python 说:

NameError: name 'i' is not defined

但是当我使用一个 URL 时它工作正常!

start_urls = ["http://example.com/category/top/page-%d/" % i for i in xrange(4)"]

Answer 1

你的 python 语法不正确，试试:

start_urls = ["http://example.com/category/top/page-%d/" % i for i in xrange(4)] + \
    ["http://example.com/superurl/top/page-%d/" % i for i in xrange(55)]

如果需要写代码生成启动请求，可以定义一个start_requests()方法而不是使用 start_urls。