python - 如何在scrapy中的中间件中使用self.crawler.engine.pause()

Question

我正在尝试从中间件暂停 scrapy 引擎(正在运行的爬虫)。

当我尝试调用 self.crawler.engine.unpause() 时，我收到此错误:

“cRetry”对象没有属性“crawler”

这是我的中间件。如何访问爬虫对象？

class cRetry(RetryMiddleware):

   errorCounter = 0


   def process_response(self, request, response, spider):
        if response.status in self.retry_http_codes:
            reason = response_status_message(response.status)
            return self._retry(request, reason, spider) or response
        elif "error" in response.body:
            self.errorCounter = self.errorCounter + 1
            if self.errorCounter >= 10:
                self.crawler.engine.pause()
                os.system("restart.sh")
                print "Reset"
                time.sleep(10)
                self.crawler.engine.unpause()
                self.errorCounter = 0
            reason ="Restart Required"
            return self._retry(request, reason, spider) or response
       ### end
        return response 

Answer 1

根据我的理解，您可以重写 __init__ 和 from_crawler 方法，类似于:

class cRetry(RetryMiddleware):

    errorCounter = 0

    def __init__(self, crawler):
        super(cRetry, self).__init__(crawler.settings)
        self.crawler = crawler

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_response(self, request, response, spider):
        # ...

__init__ 的签名实际上似乎并不重要，entry point for the main library始终是 from_crawler(cls,crawler)。这是class method并将类名作为第一个参数(然后使用它来调用构造函数)。