Python Scrapy，从子页面返回继续抓取

Question

我的蜘蛛功能在一个页面上，我需要转到一个链接并从该页面获取一些数据以添加到我的项目，但我需要从父页面转到各个页面而不创建更多项目。我将如何去做，因为根据我在文档中可以阅读的内容，我只能以线性方式进行:

  parent page > next page > next page

但我需要:

  parent page > next page
              > next page
              > next page

Answer 1

你应该返回 Request实例并在 meta 中传递 item .而且您必须以线性方式制作它并构建请求和回调链。为了实现它，您可以传递一个完成项目的请求列表，并从最后一个回调中返回一个项目:

def parse_main_page(self, response):
    item = MyItem()
    item['main_url'] = response.url

    url1 = response.xpath('//a[@class="link1"]/@href').extract()[0]
    request1 = scrapy.Request(url1, callback=self.parse_page1)

    url2 = response.xpath('//a[@class="link2"]/@href').extract()[0]
    request2 = scrapy.Request(url2, callback=self.parse_page2)

    url3 = response.xpath('//a[@class="link3"]/@href').extract()[0]
    request3 = scrapy.Request(url3, callback=self.parse_page3)

    request.meta['item'] = item
    request.meta['requests'] = [request2, request3]
    return request1

def parse_page1(self, response):
    item = response.meta['item']
    item['data1'] = response.xpath('//div[@class="data1"]/text()').extract()[0]

    return request.meta['requests'].pop(0)

def parse_page2(self, response):
    item = response.meta['item']
    item['data2'] = response.xpath('//div[@class="data2"]/text()').extract()[0]

    return request.meta['requests'].pop(0)

def parse_page3(self, response):
    item = response.meta['item']
    item['data3'] = response.xpath('//div[@class="data3"]/text()').extract()[0]

    return item

另见:

• > How can i use multiple requests and pass items in between them in scrapy python
• > Almost Asynchronous Requests for Single Item Processing in Scrapy