python - "AttributeError: ' SelectorList ' object has no attribute ' 在Scrapy Cloud中获取'"

Question

我正在使用 Scrapy 设置一个在我的笔记本电脑上运行良好的爬虫。但是当我在 scrapy cloud 上尝试这个相同的蜘蛛时，这条消息出现了:

File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/tmp/unpacked-eggs/__main__.egg/ccv_spiders/spiders/gmitem.py", line 31, in parse
    data["marque"] = caritem.css("div.make::text").get().strip().split(" ", 2)[1]
AttributeError: 'SelectorList' object has no attribute 'get'

这是我的代码:

def start_requests(self):
        for item in self.data:
            request = scrapy.Request(item['gm_url'], callback=self.parse)
            request.meta['item'] = item
            yield request

    def parse(self, response):
        item = response.meta['item']
        item['results'] = []

        for caritem in response.css("div.car-item-border"):
            data = AuctionItem()
            urllot = "https://www.website.com/img/auctions/byLot/"
            urlbase = "https://www.website.com/img/auctions/car/thumb/"
            data["marque"] = caritem.css("div.make::text").get().strip().split(" ", 2)[1]
            data["model"] = caritem.css("div.make::text").get().strip().split(" ", 2)[2]
            data["model_year"] = caritem.css("div.make::text").get().strip().split(" ", 1)[0]
            data["price_str"] = caritem.css("div.price::text").get().strip().replace(",", " ")
            if caritem.css("div.price::text").get().find("Estimate"):
                data["sold"] = True
            else:
                data["sold"] = False
            data["auction_house"] = caritem.css("div.auctionHouse::text").get().split("-", 1)[0].strip()
            data["auction_country"] = caritem.css("div.auctionHouse::text").get().rsplit(",", 1)[1].strip()
            data["auction_date"] = caritem.css("div.date::text").get().replace(",", "").strip()
            if caritem.css("div.view-auction a::attr(href)").get().find("/auction-cars/show-backup-image"):
                data["auction_url"] = caritem.css("div.view-auction a::attr(href)").get()
            else:
                data["auction_url"] = None
            data["image_urls"] = caritem.css("img.img-responsive::attr(src)").get()
            if urllot in data["image_urls"]:
                data["image_cloud"] = caritem.css("img.img-responsive::attr(src)").get().replace(urllot,"https://res.cloudinary.com/ccv/image/upload/auctions/")
                data["image_cloud"] = re.sub(r"(?<=[A-Z])/*(?=\d)", "-", data["image_cloud"])
            elif urlbase in data["image_urls"]:
                data["image_cloud"] = caritem.css("img.img-responsive::attr(src)").get().replace(urlbase, "https://res.cloudinary.com/ccv/image/upload/auctions/")
            item['results'].append(data)

        yield item

我的 Python 版本有问题吗？它在我的笔记本电脑上与 Anaconda 和 Python 3 一起工作，我不明白为什么它似乎通过“/usr/local/lib/python2.7 ...”使用 python 2.7

此外，我的 JSON 输出不显示任何结果数组。

Answer 1

一切都与库的版本有关。

get 和getall 方法均由Parsel 首次引入。 (Scrapy 的解析库)版本 1.2.0 ，如果您使用的是 Scrapy 1.5.2 或更低版本，则不会被授予。

您可以使用extract_first 和extract 作为替换，或者将Scrapy 升级到1.6+ .