node.js - 抓取 Netflix

Question

尝试以编程方式从 Netflix 中获取我最后一次查看的数据，但在登录阶段遇到了一些问题。我当前的代码只会导致 Netflix 吐回一个我们无法处理您的请求。页面:

var request = require('request').defaults({jar: true});
var cheerio = require('cheerio');

var url = "https://www.netflix.com/Login?locale=en-GB&nextpage=https%3A%2F%2Fwww.netflix.com%2FWiViewingActivity";

request(url, function (error, response, body) {
    if (!error && response.statusCode == 200) {
        var $ = cheerio.load(body);
        var authCode = $("#login-form > input").attr("value");
        request.post(url+"?email=myemail%40gmail.com&password=mypassword&RememberMe=on&authURL="+authCode, {
        }, function(err, response, body){
            console.log(body);
        });
    }
})

有什么想法吗？

令人惊讶的是，Google 上几乎没有关于Scraping Netflix 的内容。

Answer 1

想通了，需要:

1. 发送任何类型的用户代理字符串
2. 使用请求的form参数发送表单数据
3. 手动发送 cookie

这是我获取最新观看项目的最终代码:

var request = require('request').defaults({jar: true});
var cheerio = require('cheerio');

var url = "https://www.netflix.com/Login";

request(url, function (error, response, body) {
    if (!error && response.statusCode == 200) {
        var $ = cheerio.load(body);
        var authCode = $("#login-form > input").attr("value");
        request.post({url: url, 
            form: {
                "email": "email@gmail.com",
                "password": "password",
                "authURL": authCode,
                "RememberMe": "on"
            },
            headers:{
                'User-Agent': "NodeScrape"
            }
        }, function(err, response, body){
            var cookies = response.headers['set-cookie'];
            request({url: "https://www.netflix.com/WiViewingActivity", headers: {'Cookie': cookies, 'User-Agent': "NodeScrape"}}, function(error, response, body){
                var $ = cheerio.load(body);
                console.log($(".seriestitle").eq(0).text());
            });
        });
    }
})