python - 如何访问 Item Pipeline Scrapy 中的请求对象

Question

我有一个项目管道来处理价格。我在此管道中处理项目时遇到错误。但是 scrapy error 并不能说明是哪个 url 产生了错误。有什么办法可以访问管道内的请求对象

def process_item(self, item, spider):
    """
        :param self:
        :param item:
        :param spider:
    """
    print dir(spider) # No request object here...
    quit()
    if not all(item['price']):
        raise DropItem

    item['price']['new'] = float(re.sub(
        "\D", "", item['price']['new']))
    item['price']['old'] = float(re.sub(
        "\D", "", item['price']['old']))
    try:
        item['price']['discount'] = math.ceil(
            100 - (100 * (item['price']['new'] /
                          item['price']['old'])))
    except ZeroDivisionError as e:
        print "Error in calculating discount {item} {request}".format(item=item, request=spider.request) # here I want to see the culprit url...
        raise DropItem

    return item

Answer 1

您不能从 ItemPipeline 中访问响应(和 response.url)，但我认为更简单的解决方案是添加一个临时的 url 字段当你产生元素时分配，比如:

yield {...
       'url': response.url,
       ...}

可以在管道内轻松访问 url。