python - Scrapy CSRF cookie 不被接受并导致 302 重定向

Question

我对编程总体来说是新手，所以我希望这不是一个愚蠢的问题。我已经用谷歌搜索并花了过去 4 个小时试图解决这个问题，但无法解决，所以非常感谢我应该尝试解决这个问题的建议/步骤。谢谢!

这是迄今为止我对蜘蛛的了解: 从 scrapy.spider 导入 BaseSpider 从 scrapy.selector 导入 HtmlXPathSelector 从tutorial.items导入TutorialItem 从 scrapy.http 导入 FormRequest, 请求

class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']

    def parse(self, response):
        return FormRequest.from_response(response,
                    formdata={'username_or_email': '...', 'password': '...'},
                    callback=self.after_login, dont_filter = True)

    def after_login(self, response):
        print response.url

据我了解，Scrapy 会自动处理 cookie，因此 CSRF token 会通过。我在设置中将 COOKIES_ENABLED 和 COOKIES_DEBUG 设置为 True:

SPIDER_MIDDLEWARES = {'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware':     
700,}
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like 
Gecko) Chrome/29.0.1547.66 Safari/537.36"
COOKIES_ENABLED = True
COOKIES_DEBUG = True

这是调试的输出:

2013-09-27 11:11:42-0700 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Received cookies from: <200 https://
www.pinterest.com/login/>
        Set-Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; Domain=.pinteres
t.com; expires=Fri, 26-Sep-2014 18:11:46 GMT; Max-Age=31449600; Path=/
        Set-Cookie: _pinterest_sess="eJwr9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8
vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfXMyjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb
4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="; Domain=.pinterest.com; expires=Mon, 22-Sep-2014
18:11:46 GMT; Max-Age=31103999; Path=/
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Crawled (200) <GET https://www.pinte
rest.com/login/> (referer: None)
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <POST https://ww
w.pinterest.com/login/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Redirecting (302) to <GET http://www
.pinterest.com/csrf_error/> from <POST https://www.pinterest.com/login/>
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <GET http://www.
pinterest.com/csrf_error/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:44-0700 [pinterest] DEBUG: Crawled (200) <GET http://www.pinter
est.com/csrf_error/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/csrf_error/

问题是，在设置 cookie 并将其发送到登录页面后，我仍然收到 CSRF 错误并被重定向。我是否做错了什么，无法像使用浏览器一样模拟登录过程？我尝试将用户代理设置为 iPhone，得到代码 200 并且没有重定向，但 response.url 显示“https://www.pinterest.com/login/?next=/login/ ”，因此它仍然无法正确登录。

非常感谢我能得到的所有帮助。谢谢!

Answer 1

看起来非移动登录页面使用 XHR 请求来执行登录。您可以尝试深入研究 XHR 请求，也许还可以研究 javascript 代码，看看您需要做什么才能在 scrapy 中重现该请求。

但是，正如您所指出的，有一个移动登录页面可以通过更改用户代理来启用。

后一种方法的问题在于电子邮件字段是 email 而不是 username_or_email。

这是有微小变化的蜘蛛:

from scrapy.http import FormRequest
from scrapy.spider import BaseSpider


class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']
    # you can set the user agent either in the settings or the spider
    user_agent = ('Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) '
                  'AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 '
                  'Mobile/9A334 Safari/7534.48.3')

    def parse(self, response):
        data = {'email': 'XXX@xxx', 'password': 'xxx'}
        # no need for dont_filter
        return FormRequest.from_response(response, formdata=data, callback=self.after_login)

    def after_login(self, response):
        print response.url

输出:

$ scrapy runspider pinterest.py
2013-09-28 19:16:58-0400 [scrapy] INFO: Scrapy 0.16.5 started (bot: scrapybot)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-28 19:16:58-0400 [pinterest] INFO: Spider opened
2013-09-28 19:16:58-0400 [pinterest] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-28 19:17:01-0400 [pinterest] DEBUG: Crawled (200) <GET https://www.pinterest.com/login/> (referer: None)
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/> from <POST https://www.pinterest.com/login/?next=%2Flogin%2F>
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/join/discover/> from <GET http://www.pinterest.com/>
2013-09-28 19:17:10-0400 [pinterest] DEBUG: Crawled (200) <GET http://www.pinterest.com/join/discover/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/join/discover/