c# - HttpWebRequest 在代码中抛出异常但在浏览器中不抛出异常

Question

我目前正在编写一个 API，用于从销售点系统的网络界面获取数据。到目前为止，在出现这种情况之前，我在登录和生成报告以获取数据时没有遇到任何问题。

一般来说，我可以使用 follow 方法返回一个 HttpWebRequest 对象，该对象可以处理对 Web 服务器的大多数请求。

private HttpWebRequest DefaultRequestObject(string path)
{
    var request = (HttpWebRequest)WebRequest.Create(_baseUrl + path);
    request.Method = "GET";
    request.Host = _baseUrl.Substring(8); // Cut off the https://
    request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0";
    request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
    request.Headers.Add("Accept-Language", "en-US,en;q=0.5");
    request.Headers.Add("Accept-Encoding", "gzip, deflate");

    request.SendChunked = false;
    request.AllowAutoRedirect = true;

    request.ProtocolVersion = HttpVersion.Version11;

    var sp = request.ServicePoint;
    sp.Expect100Continue = false;
    var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
    prop.SetValue(sp, (byte)0, null);

    request.CookieContainer = _cookieJar;

    if (!String.IsNullOrEmpty(_cookieString))
        request.Headers.Add(HttpRequestHeader.Cookie, _cookieString);

    return request;
}

当我发送 GET 请求时，我使用以下方法:

public MgrngResponse GetContent(string serverPath, Dictionary<string, string> args)
{
    if (!serverPath.StartsWith("/"))
        serverPath = "/~pos/mgrng/" + serverPath;

    var requestString = serverPath + "?" + HttpDataFormat(args);
    var request = DefaultRequestObject(requestString);

    try
    {
        var response = (HttpWebResponse)request.GetResponse();
        var mgrngResponse = new MgrngResponse(response);

        if (!String.IsNullOrEmpty(mgrngResponse.HttpResponse.GetResponseHeader("Set-Cookie")))
            SaveMgrngResponseCookies(mgrngResponse);

        _sessionExpiration = DateTime.Now.AddMinutes(15);
        UpdateStatus(mgrngResponse.Content);

        return mgrngResponse;
    }
    catch (WebException webException)
    {
        using (WebResponse response = webException.Response)
        {
            var httpResponse = (HttpWebResponse)response;
            Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
            using (Stream data = response.GetResponseStream())
            using (var reader = new StreamReader(data))
            {
                string text = reader.ReadToEnd();
                Console.WriteLine(text);
            }
        }
        var eventArgs = new SessionUpdateEventArgs(SessionStatus.ConnectionError, "Unable to GET data");
        RaiseStatusChangeEvent(eventArgs);
        return null;
    }
}

到目前为止，这对我尝试过的所有页面都适用，但现在我遇到了一个新问题，当我尝试获取特定页面的响应时，该方法抛出一个带有 500 Internal 的 WebException服务器错误。

我使用 Fiddler 将浏览器的请求与我的请求完全匹配(显然 header 的顺序和 cookie 值除外)，但我仍然收到内部服务器错误。

以下是来自 Fiddler 的原始请求。第一个来自 Firefox，第二个来 self 的程序。

---

GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
Host: location.posprovider.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: USER=af182fda473354eb3199522726ca61c9d5516c95f165cffd63b9522726ca61c9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9fc516c950a6607ae63b9522726ca61c9; PHPSESSID=9d7f54f9a1769a3e0572745fe0db3d97
Connection: keep-alive

---

GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Host: location.posprovider.com
Cookie: PHPSESSID=16ad21c9d69fe85b3d185ab284f8620b; USER=30dca66f355e0ba89b6eda3c3e822ea24a95e5209e0f90bec94eda3c3e822ea243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e826395e520780b58edc94eda3c3e822ea2
Connection: Keep-Alive

---

我什至尝试登录到 Web 界面，然后将生成的请求字符串复制并粘贴到浏览器中，我得到了所需的数据。

有什么想法吗？

Answer 1

将您的程序请求拉入 Fiddlers composer 并逐渐消除剩余的差异。你发现这样做有什么意义？

Great advice! Apparently there's something wrong with my cookie line. I copy and pasted the cookie line from the browser's version into mine and it worked. Now I just need to resolve why my cookies aren't correct...

我同意这个诊断。谁知道服务器内部发生了什么。可能是一些脆弱的代码需要非常精确的 cookie 字符串格式。