c# - HttpClient 无法访问登录页面后面的页面

Question

[编辑 - 添加了来自 fiddler 的分析，添加了更多代码来复制身份验证 header ][编辑 - 现在使用 FormUrlEncodedContent]

我这里有一个页面:https://www.cdc.co.nz/products/list.html?cat=5201通过此处登录受密码保护:https://www.cdc.co.nz/login/

下面的代码允许我成功登录。但是，尽管使用相同的客户端，但我无法调用上述页面(401 未授权)

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

    var baseAddress = new Uri("https://www.cdc.co.nz");

    var cookieContainer = new CookieContainer();

    using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer, UseCookies = true })

    using (HttpClient client = new HttpClient(handler) { BaseAddress = baseAddress })
    {
        HttpResponseMessage response = null;

        //Let's visit the homepage to set initial cookie values
        Task.Run(async () => response = await client.GetAsync("/")).GetAwaiter().GetResult(); //200

        string urlToPost = "/login/";

        var postData = new List<KeyValuePair<string, string>>();
        postData.Add(new KeyValuePair<string, string>("username", "username"));
        postData.Add(new KeyValuePair<string, string>("password", "password"));

        HttpContent stringContent = new FormUrlEncodedContent(postData);

        client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
        client.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.9,en;q=0.8");
        client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36");

        client.DefaultRequestHeaders.Add("Origin", "https://www.cdc.co.nz");
        client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");

        client.DefaultRequestHeaders.Add("Connection", "keep-alive");
        client.DefaultRequestHeaders.Add("Host", "www.cdc.co.nz");

        client.DefaultRequestHeaders.Add("Referer", "https://www.cdc.co.nz/login/");

        cookieContainer.Add(baseAddress, new Cookie("_ga", "GA1.3.720299450.1533761418"));
        cookieContainer.Add(baseAddress, new Cookie("_gat_oldTracker", "1"));
        cookieContainer.Add(baseAddress, new Cookie("_gat", "1"));
        cookieContainer.Add(baseAddress, new Cookie("_gid", "GA1.3.1011102476.1533761418"));




//Tyler's suggestion here works! 
            //cookieContainer.Add(baseAddress, new Cookie("PHPSESSID", "value from browser login response header"));

        //Receiving 200 response for the nextline, though it returns a 302 in a browser environment
        Task.Run(async () => response = await client.PostAsync(urlToPost, stringContent)).GetAwaiter().GetResult();

        //401 response for the next line
        Task.Run(async () => response = await client.GetAsync("/products/list.html?cat=5201")).GetAwaiter().GetResult();
    }

浏览器环境的Fiddler:结果:302协议(protocol):HTTPS主办方:www.cdc.co.nz网址:/登录/

原始请求 header 浏览器环境:

POST /login/ HTTP/1.1
Host: www.cdc.co.nz
Connection: keep-alive
Content-Length: 69
Cache-Control: max-age=0
Origin: https://www.cdc.co.nz
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://www.cdc.co.nz/login/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: _ga=GA1.3.720299450.1533761418; _gid=GA1.3.1011102476.1533761418; PHPSESSID=p3jn5qqhcul59blum597mp2o41; _gat=1; _gat_oldTracker=1

Response Raw Header(Set-Cookie: PHPSESSID=oh7in7n5pjbkrkng4qwwwn22uaq951 是我感兴趣的)在浏览器环境下:

HTTP/1.1 302 Found
Date: Thu, 09 Aug 2018 00:51:11 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.25
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=oh7in7n5pjbkrkng4qwwwn22uaq951 <-------- Needed in subsequent Request headers to not 401.
Location: https://www.cdc.co.nz/home/news.html
Content-Length: 0
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

HttpClient 的 Fiddler:结果:200协议(protocol):HTTPS主办方:www.cdc.co.nz网址:/登录/

HttpClient 环境中的原始 header :

    GET /login/ HTTP/1.1
    Host: www.cdc.co.nz
    Connection: keep-alive
    Upgrade-Insecure-Requests: 1
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
    Referer: https://www.cdc.co.nz/home/my-account/
    Accept-Encoding: gzip, deflate, br
    Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
    Cookie: _ga=GA1.3.720299450.1533761418; _gid=GA1.3.1011102476.1533761418; _gat=1; _gat_oldTracker=1; PHPSESSID=sdjm7r2jge751jo39mkesqnfl6

HttpClient 环境中的原始响应 header (注意这里怎么没有 Set-Cookie header /值？):

HTTP/1.1 200 OK
Date: Thu, 09 Aug 2018 01:11:14 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.25
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Content-Length: 5668

回答

添加额外的 KV 对(甚至没有指定其他不必要的细节)现在可以使代码正常工作:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

    var baseAddress = new Uri("https://www.cdc.co.nz");

    using (HttpClient client = new HttpClient() { BaseAddress = baseAddress })
    {
        HttpResponseMessage response = null;

        //Let's visit the homepage to set initial cookie values
        Task.Run(async () => response = await client.GetAsync("/")).GetAwaiter().GetResult(); //200

        string urlToPost = "/login/";

        var postData = new List<KeyValuePair<string, string>>();
        postData.Add(new KeyValuePair<string, string>("username", "username"));
        postData.Add(new KeyValuePair<string, string>("password", "password"));
        postData.Add(new KeyValuePair<string, string>("returnUrl", "/login/")); <----- To simulate the browser
        postData.Add(new KeyValuePair<string, string>("service", "login")); <----- To simulate the browser

        HttpContent stringContent = new FormUrlEncodedContent(postData);

        //Receiving 200 response for the nextline, though it returns a 302 in a browser environment
        Task.Run(async () => response = await client.PostAsync(urlToPost, stringContent)).GetAwaiter().GetResult();

        //200 response now
        Task.Run(async () => response = await client.GetAsync("/products/list.html?cat=5201")).GetAwaiter().GetResult();
    }

Answer 1

尝试像浏览器那样添加隐藏的表单值

    var postData = new List<KeyValuePair<string, string>>();
    postData.Add(new KeyValuePair<string, string>("username", "username"));
    postData.Add(new KeyValuePair<string, string>("password", "password"));
    postData.Add(new KeyValuePair<string, string>("returnUrl", "/login/"));
    postData.Add(new KeyValuePair<string, string>("service", "login"));