python - Scrapy:如何从splash中获取cookie

Question

我试图从启动请求中获取 cookie，但我不断收到错误消息。

这是我正在使用的代码:

class P2PEye(scrapy.Spider):
    name = 'p2peyeSpider'
    allowed_domains = ['p2peye.com']
    start_urls = ['https://www.p2peye.com/platform/h9/']

    def start_requests(self):
        script = '''
        function main(splash)
          local url = splash.args.url
          assert(splash:go(url))
          assert(splash:wait(0.5))
          return {
            cookies = splash:get_cookies(),
          }
        end
        '''
        for url in self.start_urls:
            yield SplashRequest(url, callback=self.parse, endpoint='render.html',args={'wait': 1, 'lua_source': script})

    def parse(self, response):
        print(response.request.headers.getlist('Set-Cookie'))
        print(response.cookiejar)

这是我的settings.py

SPLASH_URL = 'http://127.0.0.1:8050'
CRAWLERA_ENABLED= False
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {'scrapy_splash.SplashDeduplicateArgsMiddleware': 100 }
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
COOKIES_ENABLED = True
COOKIES_DEBUG = True
SPLASH_COOKIES_DEBUG = True

response.request.headers.getlist('Set-Cookie')的结果是[]，并且 response.cookiejar 出现错误:AttributeError: 'SplashTextResponse' object has no attribute 'cookiejar'。那么如何获取 cookie 而不会导致错误呢？

Answer 1

要访问response.cookiejar，您需要返回SplashJsonResponse

尝试在 Lua 脚本中返回额外的字段:

script = '''
        function main(splash)
          local url = splash.args.url
          assert(splash:go(url))
          assert(splash:wait(0.5))
          local entries = splash:history()
          local last_response = entries[#entries].response
          return {
            url = splash:url(),
            headers = last_response.headers,
            http_status = last_response.status,
            cookies = splash:get_cookies(),
            html = splash:html(),
          }
        end
        '''