c# - HTTPClient 缓冲区超过 2G；无法向缓冲区写入更多字节

Question

正在成为 Walmart API 传奇的最新消息。我正在提交调用以使用 HttpClient 获取清关项目列表。其他请求工作正常，但是这个请求太大了，它破坏了 HTTPRequest 缓冲区。 同样奇怪的是，这是请求缓冲区错误而不是响应错误，因为请求只是 URL。

异常信息:

 Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647. (System.Net.Http)

at System.Net.Http.HttpContent.LimitMemoryStream.CheckSize(Int32 countToAdd)\r\n at System.Net.Http.HttpContent.LimitMemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n at System.Net.Http.StreamToStreamCopy.TryStartWriteSync(Int32 bytesRead)\r\n at System.Net.Http.StreamToStreamCopy.StartRead()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at Wal_Mart_Crawler.NET_IO.<walMart_Special_Feed_Lookup>d__4.MoveNext() in c:\\users\\user\\documents\\visual studio 2015\\Projects\\Wal-Mart_Inventory_Tracker\\Wal-Mart_Crawler\\NET_IO.cs:line 65\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at Wal_Mart_Crawler.Special_Feeds.d__2.MoveNext() in c:\users\user\documents\visual studio 2015\Projects\Wal-Mart_Inventory_Tracker\Wal-Mart_Crawler\Special_Feeds.cs:line 22

(好奇的是，第 22 行是):

 API_Json_Special_Feeds.RootObject Items = await net.walMart_Special_Feed_Lookup(specialFeedsURLs[i].Replace("{apiKey}", Wal_Mart_Crawler.Properties.Resources.API_Key_Walmart));
 //which uses HttpClient to call the API

起初，当我看到“已配置”时，我的眼睛闪了一下，因为那意味着我可以更改它，对吗？毕竟我在 x64 中运行 - 可能能够突破 2G 不知道如何。

阅读 SO，发现我应该禁用流式传输。尝试过:

 var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);

...没有布埃诺。

为 WebRequest 找到了一个 AllowReadStreamBuffering 设置 - 似乎找不到 HttpClient 的设置。

我无法控制我获得了多少数据——那是在沃尔玛。我也受限于如何处理响应流，因为它会直接反序列化:

   var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
   return JsonConvert.DeserializeObject<API_Json_Special_Feeds.RootObject>(result);

因此，即使我可以分解响应，我也会被卡住，因为我认为我无法提供部分数据以进行反序列化。

问题:如何将缓冲区增加到 2G 以上或以其他方式避免缓冲区大小超出异常？

我希望这是另一个简单的解决方法，我疲惫无知的大脑就是想不通。一如既往 - 衷心感谢您抽出时间并提前感谢您提供的任何帮助。

响应 header :

 StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
 X-Mashery-Responder: prod-j-worker-us-west-1c-63.mashery.com
 transfer-encoding: chunked
 Connection: keep-alive
 Date: Sat, 10 Sep 2016 00:57:38 GMT
 Server: Mashery
 Server: Proxy
 Content-MD5: BvJMDJiZPUvmAxxmwKGSog==
 Content-Type: application/json; charset=utf-8
 Last-Modified: Fri Sep 09 15:31:08 PDT 2016
}

Answer 1

这样的东西行得通吗？

using (var http = new HttpClient())
using (var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
using (StreamReader sr = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
    var serializer = new JsonSerializer();               
    ITraceWriter tw = new MemoryTraceWriter();
    serializer.TraceWriter = tw; 
    var obj = (API_Json_Special_Feeds.RootObject)serializer.Deserialize(sr, typeof(API_Json_Special_Feeds.RootObject));
    // Stop and inspect the tracewriter object here to diagnose 
    return obj;                
}

编辑:通过包含跟踪编写器，您可以尝试并确定为什么反序列化的行为不符合您的预期。 JSON 反序列化往往会悄无声息地丢弃它不知道如何处理的信息，并导致一个空的或稀疏填充的对象。 tracewriter 将帮助诊断这些问题。 Example