.net - 了解KeepAlive模式下的HttpWebRequest

Question

我知道这个话题已经讨论过很多次了，但我需要了解如何以正确的方式编写代码。
我在协议(protocol)版本 HTTP 1.1 中多次使用相同的 HttpWebRequest(到相同的 url)。

Method = "POST"
KeepAlive = True

但每次我需要发送不同的请求，并得到不同的响应。
(注意。下一个代码不正确并引发异常)

Private Sub SendHttpWebReq()
    Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)
    httpWebReq.Method = "POST"
    httpWebReq.KeepAlive = True
    httpWebReq.ContentType = "application/x-www-form-urlencoded"
    Dim myRequestString As New List(Of String) From {"abc", "def"}
    Dim ContentList As New List(Of String)
    For a = 0 To 1
        Dim inputData As String = MyRequestString(a)
        Dim postData As String = "firstone" + ChrW(61) + inputData
        Dim encoding As New System.Text.ASCIIEncoding()
        Dim byteData As Byte() = encoding.GetBytes(postData)
        httpWebReq.ContentLength = byteData.Length
        Dim newStream As IO.Stream = httpWebReq.GetRequestStream()
        newStream.Write(byteData, 0, byteData.Length)
        newStream.Flush()
        newStream.Dispose()
        Dim Response As Net.WebResponse = httpWebReq.GetResponse()
        Dim ResponseStream As Io.Stream = Response.GetResponseStream()
        Dim Content = New Io.MemoryStream()
        ResponseStream.CopyTo(Content)
        Response.Close()
        Response.Dispose()
        ResponseStream.Flush()
        ResponseStream.Dispose()
        ContentList.Add(System.Text.Encoding.UTF8.GetString(Content.ToArray))
        Content = Nothing
    Next
End Sub

当我运行代码时，我第一次得到了正确的响应，但是当我尝试重用 HttpWebRequest 时，在这一行抛出了一个异常:

httpWebReq.ContentLength = byteData.Length

异常是写入开始后无法设置此属性
搜索，我找到了这个话题:
Am I able to reuse a HttpWebRequest?
在那里解释说要重用 HttpWebRequest，必须关闭 Stream 和 WebResponse，我做到了，释放了资源。
同样在这个主题中，它解释了同样的事情:
Reusing HttpWebRequest Object
但在另一个主题中:
This property cannot be set after writing has started! on a C# WebRequest Object
一位成员说不可能重用 HttpWebRequest。
我在重用和创建一个新的之间感到困惑，我需要了解什么 KeepAlive它指的是:连接，还是请求？
我想当我执行这条指令时:

Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)

我应该创建一个 HttpWebRequest 类的实例，但我应该用这个指令建立连接:

Dim newStream As IO.Stream = httpWebReq.GetRequestStream()

我对么？

Answer 1

这是我认为需要澄清的地方，因为该声明可能被认为具有误导性，其措辞方式如下:

1 WebRequest => 1 WebResponse. You can't change anything in a WebRequest once it has been initialized.

这在原则上仍然有效，但初始化的术语可能会令人困惑。更好的定义是:

You can't change any parameter of a WebRequest after the request has been issued and a WebResponse has been returned, until after the WebResponse is closed (disposed).

经过 WebResponse已经返回一个结果，它可以被关闭 - 显式或隐式(在 Using 块中) - 你可以请求另一个，根据需要修改 WebRequest参数(例如，将方法从 POST 更改为 GET)。
此外，在请求新的 WebResponse 之前，必须重新初始化 WebRequest。如果你不这样做，它只会回退到它的默认值。
我在下面发布的代码是一个经典上下文(Web 登录请求)的示例，当 WebRequest必须在同一过程中多次重新初始化，以接收不确定数量的 WebResponse，直到到达目标地址(或登录页面)。
这或多或少是架构:

                  --------------
(GET or POST)     | WebRequest |       (Method is POST)
      |---------> | GET/(POST) | <-----------| <-------------- |
      |           --------------             |                 |
      |                 |                    |                 |
--------------    ---------------    ------------------   --------------
|    New     |    | WebResponse |--> | LogIn Required |-->|   LogIn    |
|  Location  |    ---------------    ------------------   |   Address  |
| (Referer   |          |                                 --------------
|    Set)    |          |
--------------     (Set Cookies)
      |                 |
      |           ---------------
      |           |    LogIn    |
 Redirection <----|     OK      |---NO---|
                  ---------------        |
                        |                |
                       YES               |
                   (Set Cookies)         |
                        |             Request
                  ---------------     Denied
                  |  Response   |        |
                  |    URI      |        |
                  ---------------        |
                        |                |
                       EXIT <------------|
                        |

请注意，发出 WebRequest，如果访问请求的资源 URI 需要身份验证，则服务器可能不会以 StatusCode 回答。 302 (Found) , 301 (Moved)或 303 (Redirected) ，它可能只是将 StatusCode 设置为 200 (OK) .重定向是隐式的，因为设置了“位置” header ，或者如果是 WebForm 登录，则检索到的 Html 页面包含重定向。
无论如何，在检测到重定向后，必须遵循新的重定向位置到达目的地。一个重定向可能包含一个或多个 Hops ，这通常必须手动处理(以验证我们是否被发送到我们真正想去的地方)。

关于keep-alive标题 .
一个 keep-alive header 由客户端和/或服务器设置为 提示 成立的对口 连接应该保持开放，至少一段时间，因为与当前交易相关的其他资源被交换的可能性很高。
这可以防止创建大量昂贵的连接。
此设置常见于 Http和 Ftp请求，这是 Http 1.1 中的标准.
Hypertext Transfer Protocol (HTTP) Keep-Alive Header (IETF)
Compatibility with HTTP/1.0 Persistent Connections (IETF)
需要记住的是 keep-alive header 指的是 连接 已经通过 WebRequest 建立的，而不是 WebRequest 本身。当一个连接被创建并指定它应该保持打开时，后续的请求应该维护 connection: keep-alive符合协议(protocol)的 header 。
在 .NET 中 HttpWebRequest ，设置 KeepAlive属性(property)到 False , 相当于设置 connection: close标题。
但是，管理连接和进程被授予访问权限的连接池的逻辑由 ServicePointManager 管理。 ，使用 ServicePoint作为每个连接请求的引用。
因此，WebRequest 可以指定它要求创建的 Connection 应该保持打开状态(因为它需要被重复使用更多次)，但是 Connection 背后的真正逻辑，它是如何建立、维护和管理的，则位于其他地方。
在 HTTP 2.0(StackOverflow/StackExchange 使用此协议(protocol))中，完全忽略了保持事件设置，正是出于这个原因。连接逻辑在更高的、独立的级别进行管理。

(...) when you call 1 new HttpWebRequest every 3 seconds, every 10seconds, or every 60 seconds? What's the difference when i send thoserequests with True or False?

您设置了 KeepAlive属性(property)到 提示 建立连接的连接管理器应该保持打开状态，因为您知道它将被重新使用。但是， ServicePointManager如果使用的协议(protocol)提供此设置，并且在管理连接池复合体的内部逻辑所施加的限制内，则远程服务器将遵守该请求。
它应该设置在 HTTP 1.0 , 这是 HTTP 1.1 中的默认设置, 在 HTTP 2.0 中被忽略.
由于在建立连接之前您无法知道将使用哪种协议(protocol)，因此通常将其设置为 keep-alive ，因为到请求资源的路由中的某些设备(特别是代理)可能需要明确设置此设置(阅读有关代理及其行为的 IETF 文档)。

在这个例子中，一个WebRequest在一个Loop中被反复初始化，并且每次都释放底层的WebResponse，直到出现一个StatusCode 200 (OK)收到或请求被拒绝或我们被重定向太多次(取消 token 在这里也可能有用)。
在示例中，主要方法旨在以这种方式调用:

Public Async Sub SomeMethodAsync()
    LoginParameters = New LoginObject() With {
        .CookieJar = New CookieContainer,
        .LogInUrl = "[Some IP Address]",
        .Credentials = New Dictionary(Of String, String)
    }
    LoginParameters.Credentials.Add("UserName", "[Username]")
    LoginParameters.Credentials.Add("Email", "[email]")
    LoginParameters.Credentials.Add("Password", "[Password]")

    LoginParameters = Await HttpLogIn(LoginParameters)
7End Sub

LoginParameters必须保留对象，因为引用了 CookieContainer ，其中包含认证后收到的 Cookie。当一个新的 WebRequest 被初始化时，这些 Cookies 被传递到服务器，作为请求的凭证已经被认证的“证明”。请注意，这些 Cookie 会在一段时间后过期(这些 Cookie 在发出新的 WebRequest 时会“刷新”，除非 Session 有时间限制)。如果是这种情况，登录过程会自动重复。

Imports System.Net
Imports System.Net.Security
Imports System.IO
Imports System.Security
Imports System.Security.Cryptography
Imports System.Security.Cryptography.X509Certificates
Imports System.Text

Public LoginParameters As LoginObject

Public Class LoginObject
    Public Property LogInUrl As String
    Public Property ResponseUrl As String
    Public Property Credentials As Dictionary(Of String, String)
    Public Property StatusCode As HttpStatusCode
    Public Property CookieJar As New CookieContainer()
End Class

Public Async Function HttpLogIn(LogInParameters As LoginObject) As Task(Of LoginObject)
    Dim httpRequest As HttpWebRequest
    Dim StatusCode As HttpStatusCode
    Dim MaxHops As Integer = 20

    ' Windows 7 (.Net 4.5.1+ required):     
    'ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12

    ' Windows 10 (.Net 4.5.1+ required):    
    ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault

    'If needed or for testing
    'ServicePointManager.ServerCertificateValidationCallback = AddressOf CertificateValidation

    httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)

    Try
        HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)
        Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
            StatusCode = httpResponse.StatusCode
        End Using

        If StatusCode = HttpStatusCode.OK OrElse StatusCode = HttpStatusCode.NoContent Then
            'POST Parameters are URLEncoded and the encoded strings converted to a Byte array of UTF8 chars
            Dim EncodedParameters As Byte() = HTTP_EncodePOSTParameters(LogInParameters.Credentials)

            httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)
            httpRequest.Method = WebRequestMethods.Http.Post
            httpRequest.ContentType = "application/x-www-form-urlencoded"
            httpRequest.ContentLength = EncodedParameters.Length
            HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)

            Using stream As Stream = Await httpRequest.GetRequestStreamAsync()
                stream.Write(EncodedParameters, 0, EncodedParameters.Length)
            End Using

            Dim Hops As Integer = 0
            Dim Referer As String = LogInParameters.LogInUrl
            Dim LastHttpMethod As String = httpRequest.Method

            Do
                'Evaluate Authentication redirect or page moved
                Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
                    StatusCode = httpResponse.StatusCode
                    LogInParameters.ResponseUrl = URIFromResponseLocation(httpResponse).ToString()
                End Using

                If (StatusCode = HttpStatusCode.Moved) OrElse
                   (StatusCode = HttpStatusCode.Found) OrElse
                   (StatusCode = HttpStatusCode.RedirectMethod) OrElse
                   (StatusCode = HttpStatusCode.RedirectKeepVerb) Then

                    httpRequest = WebRequest.CreateHttp(LogInParameters.ResponseUrl)
                    HTTP_RequestHeadersInit(httpRequest, Referer, LogInParameters.CookieJar)
                    If StatusCode = HttpStatusCode.RedirectKeepVerb Then
                        httpRequest.Method = LastHttpMethod
                    Else
                        LastHttpMethod = httpRequest.Method
                    End If
                End If

                If (CType(StatusCode, Integer) > 320) OrElse Hops >= MaxHops Then
                    Exit Do
                End If
                Hops += 1
            Loop While (StatusCode <> HttpStatusCode.OK)

            If StatusCode = HttpStatusCode.OK Then
                LogInParameters.CookieJar = httpRequest.CookieContainer
            End If
        End If

    Catch exW As WebException
        StatusCode = If(exW.Response IsNot Nothing,
                        CType(exW.Response, HttpWebResponse).StatusCode,
                        CType(exW.Status, HttpStatusCode))

    Catch exS As System.Exception
        StatusCode = CType(WebExceptionStatus.RequestCanceled, HttpStatusCode)

    Finally
        ServicePointManager.ServerCertificateValidationCallback = Nothing
    End Try

    LogInParameters.StatusCode = StatusCode
    Return LogInParameters
End Function

Private Sub HTTP_RequestHeadersInit(ByRef httpReq As HttpWebRequest, Referer As String, CookiesJar As CookieContainer)
    httpReq.Date = DateTime.Now
    httpReq.CookieContainer = CookiesJar
    httpReq.KeepAlive = True
    httpReq.ConnectionGroupName = Guid.NewGuid().ToString()
    httpReq.AllowAutoRedirect = False
    httpReq.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
    httpReq.ServicePoint.Expect100Continue = False
    httpReq.Referer = Referer
    httpReq.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"
    httpReq.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    httpReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.9,en;q=0.5")
    httpReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
    httpReq.Headers.Add(HttpRequestHeader.CacheControl, "no-cache")
End Sub

Private Function HTTP_EncodePOSTParameters(PostParameters As Dictionary(Of String, String)) As Byte()
    Dim Encoder As New System.Text.UTF8Encoding()
    Dim CredentialValues As New StringBuilder()
    Dim _first As Boolean = True

    For Each CurrentKeyPair As KeyValuePair(Of String, String) In PostParameters
        If _first = False Then CredentialValues.Append("&")
        CredentialValues.AppendFormat("{0}={1}", WebUtility.UrlEncode(CurrentKeyPair.Key),
                                                 WebUtility.UrlEncode(CurrentKeyPair.Value))
        _first = False
    Next

    Return Encoder.GetBytes(CredentialValues.ToString())
End Function

Private Function URIFromResponseLocation(Response As HttpWebResponse) As System.Uri
    Dim uri As Uri
    Dim Location As String = Response.Headers("Location")

    Try
        If uri.IsWellFormedUriString(Location, UriKind.Absolute) Then
            uri = New Uri(Location, UriKind.Absolute)
        Else
            Dim HostUri As String = Response.ResponseUri.GetComponents(UriComponents.SchemeAndServer,
                                                                        UriFormat.Unescaped) + Location
            uri = If(uri.IsWellFormedUriString(HostUri, UriKind.Absolute),
                        New Uri(HostUri),
                        New Uri(Response.ResponseUri.GetComponents(UriComponents.Scheme, UriFormat.Unescaped) +
                                Response.ResponseUri.Host + Location))
        End If
    Catch ExceptionOnInvalidUri As Exception
        uri = New Uri(Location, UriKind.Relative)
    End Try

    Return uri
End Function

Private Function CertificateValidation(sender As Object,
                                       CACert As X509Certificate,
                                       CAChain As X509Chain,
                                       PolicyErrors As SslPolicyErrors) As Boolean

    'This method, as it is, accepts a Server certificate in any case
    'It could be eventually adapted to refuse a connection (returning false) 
    'if the certificate is invalid, expired or from a untrusted path
    If (PolicyErrors = SslPolicyErrors.None) Then Return True

    'If a Certificated must be added to the Chain, uncomment the code below,
    'selecting a Certificate in the Local (or other) Storage
    'Dim MyCert As X509Certificate2 = New X509Certificate2("[localstorage]/[ca.cert]")
    'CAChain.ChainPolicy.ExtraStore.Add(MyCert)

    'CAChain.Build(MyCert)
    'For Each CACStatus As X509ChainStatus In CAChain.ChainStatus
    '    If (CACStatus.Status <> X509ChainStatusFlags.NoError) And
    '       (CACStatus.Status <> X509ChainStatusFlags.UntrustedRoot) Then
    '        Return False
    '    End If
    'Next

    Return True
End Function