python - Scrapy - 抓取取所有元素而不是 1 项

Question

我需要抓取所有元素，但只有 1 个元素是抓取的。我的代码之前工作正常，但是当我将它转移到其他代码相同的元素时，我不知道为什么会发生这种情况

我需要根据 start_url 中的页面大小获取所有元素

这是我的工作代码

class HmSalesitemSpider(scrapy.Spider):
    name = 'HM_salesitem'
    allowed_domains = ['www2.hm.com']
    start_urls = ['https://www2.hm.com/en_us/sale/shopbyproductladies/view- 
all.html?sort=stock&image-size=small&image=stillLife&offset=0&page- 
size=3002']

def parse(self, response):  
    for product_item in response.css('li.product-item'):
        url = "https://www2.hm.com/" + product_item.css('a::attr(href)').extract_first() 
    yield scrapy.Request(url=url, callback=self.parse_subpage)

def parse_subpage(self, response):
    item = {
    'title': response.xpath("normalize-space(.//h1[contains(@class, 'primary') and contains(@class, 'product-item-headline')]/text())").extract_first(),
    'sale-price': response.xpath("normalize-space(.//span[@class='price-value']/text())").extract_first(), 
    'regular-price': response.xpath('//script[contains(text(), "whitePrice")]/text()').re_first("'whitePrice'\s?:\s?'([^']+)'"),
    'photo-url': response.css('div.product-detail-main-image-container img::attr(src)').extract_first(),
    'description': response.css('p.pdp-description-text::text').extract_first()

    }   
    yield item

请帮助。谢谢

Answer 1

看来你的缩进有问题。将屈服请求移至 for 循环:

def parse(self, response):  
    for product_item in response.css('li.product-item'):
        url = "https://www2.hm.com/" + product_item.css('a::attr(href)').get() 
        yield scrapy.Request(url=url, callback=self.parse_subpage)

或者这个有点清版:

def parse(self, response):  
    for link in response.css('li.product-item a::attr(href)').extract():
        yield response.follow(link, self.parse_subpage)