python - Scrapy 爬虫没有完成解析函数中的所有循环

Question

我的爬虫中有这段代码

class StackSpider(InitSpider):
    name = 'stack'
    allowed_domains = ['sitepoint.com']
    start_urls = ["http://www.sitepoint.com"]
    start_page = "http://www.sitepoint.com"
    item = StackItem()

    def init_request(self):

        return Request(url=self.start_page, callback=self.parse)

    def parse(self, response):

        hxs = HtmlXPathSelector(response)
        sites = hxs.select('//div[@class="headline_area"]')
        items = []


        ivar = 1
        for site in sites[:5]:
            item = StackItem()
            log.msg(' LOOP' +str(ivar)+ '', level=log.ERROR)
            item['title'] ="yoo ma"
            request =  Request("http://www.sitepoint.com/getting-to-know-css3-selectors-structural-pseudo-classes/",  callback=self.test1)
            request.meta['item'] = item
            ivar = ivar + 1
            yield request


    def test1(self, response):
        log.msg('  LOOP 2 \n', level=log.ERROR)
        item = response.meta['item']
        item['desc'] = "test4"
        return item

我按照 documentation 做的但它只适用于一个循环。我的意思是我只能在登录屏幕上看到

LOOP1
LOOP2

应该重复3次

我尝试了 return 和 yield 的不同组合，所以

1. return request 和return item 给出输出 LOOP1 LOOP2
2. yield request and return item 给出输出 LOOP1 LOOP1 LOOP1 LOOP2
3. yield request 和 yield item 给出输出 LOOP1 LOOP1 LOOP1 LOOP2
4. return request and yield item 给出输出 LOOP1 LOOP2

我怎样才能得到 LOOP 1 LOOP2 LOOP1 LOOP2 AND so on

Answer 1

问题出在你的循环上

for site in sites[:5]:

您正在多次循环请求 1 个相同的 URL。

Scrapy 默认过滤相同的请求并忽略它们。

如果你想多次请求同一个 URL，你需要设置 dont_filter=True

            request = Request("http://www.sitepoint.com/getting-to-know-css3-selectors-structural-pseudo-classes/",
            dont_filter=True,
            callback=self.test1)

那么应该重复3次