html - 零碎的 : How to scrape

Question

我正在学习如何使用 scrapy api 进行抓取。

我想将文本抓取到 <h2 class > 中和链接到 <a href >但它不起作用(附件)

我试图提取 <a > 中的文本标签

import scrapy

class PriceSpider(scrapy.Spider):
    name = "annonce"  #name of spider

    def start_requests(self):
        urls = [
            'https://www.leboncoin.fr/ventes_immobilieres/offres/ile_de_france/?th=1',

        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        for annonce in response.css('section.tabsContent li').extract():
            yield{
                'title':annonce.css('a ::title').extract_first(),
                }

Answer 1

试一试。您的 CSS 选择器存在严重缺陷。

import scrapy

class PriceSpider(scrapy.Spider):
    name = "annonce"  #name of spider

    def start_requests(self):
        urls = [
            'https://www.leboncoin.fr/ventes_immobilieres/offres/ile_de_france/?th=1',

        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        for annonce in response.css('.list_item'):
            yield{
                'link':annonce.css('::attr(href)').extract_first(),
                'title':annonce.css('.item_title::text').extract_first().strip(),
                }

还有一件事。打开您的 settings.py 文件并制作它:

ROBOTSTXT_OBEY = False