python - 如何使用请求库从基本 url 抓取到 scrapy 中的另一个 url

Question

import scrapy
from scrapy.selector import Selector
from scrapy.contrib.spiders import CrawlSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request


class NextlinkSpider(scrapy.Spider):
    name = 'nextlink'
    allowed_domains = ['www.shanazrafiq.com/']
    start_urls = ['https://www.shanazrafiq.com/']

    def parse(self, response):
        yield Request(url='https://www.shanazrafiq.com/p/recipes.html',callback = self.parse_dir_contents) 

错误:AttributeError:“NextlinkSpider”对象没有属性“parse_dir_contents”

试图从基本 url 抓取到另一个 url

我想通过从第一页导航来获取食谱页面标签内容，但无法调用回调函数，该回调函数说我的属性在这样做时未找到错误。帮我解决这个问题。提前致谢

def parse_dir_contents(self, response): # second function to be called.
    sel = Selector(response)
    title_name=sel.css("div.widget HTML h2::text").extract()
    print title_name

Answer 1

根据您提供的错误消息猜测，您的 NextlinkSpider 类没有 parse_dir_contents 方法。我猜你已经在类之外(或其他地方)定义了一个同名的函数。

我建议尝试将这两者“粘合”在一起:

class NextlinkSpider(scrapy.Spider):
    name = 'nextlink'
    allowed_domains = ['www.shanazrafiq.com/']
    start_urls = ['https://www.shanazrafiq.com/']

    def parse(self, response):
        yield Request(url='https://www.shanazrafiq.com/p/recipes.html',callback = self.parse_dir_contents)

    def parse_dir_contents(self, response): # second function to be called.
       sel = Selector(response)
       title_name=sel.css("div.widget HTML h2::text").extract()
       print title_name

代码没有变化(缩进除外)