python - Scrapy 分页不起作用并优化了蜘蛛

Question

请帮助我优化我的 scrapy 蜘蛛。特别是下一页分页不起作用。有很多页，每页有 50 项。我在 parse_items 中捕获第一页 50 个项目(链接)，下一页项目也在 parse_items 中废弃。

import scrapy
from scrapy import Field
from fake_useragent import UserAgent 

class DiscoItem(scrapy.Item):
    release = Field()
    images = Field()


class discoSpider(scrapy.Spider):
    name = 'myspider'
    allowed_domains = ['discogs.com']
    query = input('ENTER SEARCH MUSIC TYPE : ')
    start_urls =['http://www.discogs.com/search?q=%s&type=release'%query]
   custome_settings = {
'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36",

   'handle_httpstatus_list' : [301,302,],
    'download_delay' :10}

    def start_requests(self):
        yield scrapy.Request(url=self.start_urls[0], callback=self.parse)   

    def parse(self, response):
        print('START parse \n')
        print("*****",response.url)

        #next page pagination 
        next_page =response.css('a.pagination_next::attr(href)').extract_first()
       next_page = response.urljoin(next_page)
       yield scrapy.Request(url=next_page, callback=self.parse_items2)

        headers={}
        for link in response.css('a.search_result_title ::attr(href)').extract():

        ua = UserAgent()# random user agent
        headers['User-Agent'] = ua.random
        yield scrapy.Request(response.urljoin(link),headers=headers,callback=self.parse_items) 


    def parse_items2(self, response):
        print('parse_items2 *******', response.url)
        yield scrapy.Request(url=response.url, callback=self.parse)  

    def parse_items(self,response):

        print("parse_items**********",response.url)
        items = DiscoItem()
        for imge in response.css('div#page_content'):
            img = imge.css("span.thumbnail_center img::attr(src)").extract()[0]
            items['images'] = img
            release=imge.css('div.content a ::text').extract()
            items['release']=release[4]
            yield items

Answer 1

当我尝试运行您的代码时(在修复了许多缩进、拼写和字母大小写错误之后)，这一行显示在 scrapy 的日志中:

2018-03-05 00:47:28 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.discogs.com/search/?q=rock&type=release&page=2> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)

Scrapy 默认情况下会过滤重复请求，而您的 parse_items2() 方法除了创建重复请求外什么也不做。我看不出该方法存在的任何原因。

您应该做的是将 ˙parse()` 方法指定为请求的回调，并避免使用不执行任何操作的额外方法:

yield scrapy.Request(url=next_page, callback=self.parse)