c# - 使用 Parallel.ForEach 循环时的连接问题

Question

我有一个foreach 循环，它负责执行一组特定的语句。其中一部分是将图像从 URL 保存到 Azure 存储。我必须为大量数据执行此操作。为了实现相同的目的，我将 foreach 循环转换为 Parallel.ForEach 循环。

 Parallel.ForEach(listSkills, item =>
 {
     // some business logic
     var b = getImageFromUrl(item.Url);
     Stream ms = new MemoryStream(b);

     saveImage(ms);
     // more business logic
 });

 private static byte[] getByteArray(Stream input)
 {
   using (MemoryStream ms = new MemoryStream())
   {
     input.CopyTo(ms);
     return ms.ToArray();
   }
 }

 public static byte[] getImageFromUrl(string url)
 {
    HttpWebRequest request = null;
    HttpWebResponse response = null;
    byte[] b = null;
    request = (HttpWebRequest)WebRequest.Create(url);
    response = (HttpWebResponse)request.GetResponse();
    if (request.HaveResponse)
    {
      if (response.StatusCode == HttpStatusCode.OK)
       {
          Stream receiveStream = response.GetResponseStream();
          b = getByteArray(receiveStream);
       }
    }

    return b;
 }

 public static void saveImage(Stream fileContent)
 {
  fileContent.Seek(0, SeekOrigin.Begin);
  byte[] bytes = getByteArray(fileContent);
  var blob = null;
  blob.UploadFromByteArrayAsync(bytes, 0, bytes.Length).Wait();
 }

尽管有些情况下我会遇到以下错误并且无法保存图像。

An existing connection was forcibly closed by the remote host.

同时分享 StackTrace :

   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Security.SslStream.<FillBufferAsync>d__183`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Net.Security.SslStream.<ReadAsyncInternal>d__181`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.Read(Span`1 buffer)
   at System.Net.Http.HttpConnection.Read(Span`1 destination)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.Read(Span`1 buffer)
   at System.Net.Http.HttpBaseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at Utilities.getByteArray(Stream input) in D:\repos\SampleProj\Sample\Helpers\CH.cs:line 238
   at Utilities.getImageFromUrl(String url) in D:\repos\SampleProj\Sample\Helpers\CH.cs:line 178

我猜这可能是因为我没有使用锁？我不确定是否在 Parallel.ForEach 循环中使用锁。

Answer 1

根据 another question on stackoverflow , 以下是 现有连接被远程主机强行关闭的潜在原因。:

• You are sending malformed data to the application (which could include sending an HTTPS request to an HTTP server)
• The network link between the client and server is going down for some reason
• You have triggered a bug in the third-party application that caused it to crash
• The third-party application has exhausted system resources

由于只有部分请求受到影响，我认为我们可以排除第一个请求。当然，这可能是网络问题，在这种情况下，这会不时发生，具体取决于您和服务器之间的网络质量。

除非您从其他用户那里发现 AzureStorage 错误的迹象，否则您的调用很可能同时消耗过多的远程服务器资源(连接/数据)。服务器和代理对它们可以同时处理的连接数量有限制(尤其是来自同一台客户端机器)。

根据 listSkills 列表的大小，您的代码可能会发起大量并行请求(尽可能多的线程池)，可能会淹没服务器。

您至少可以像这样使用 MaxDegreeOfParallelism 来限制并行任务启动的数量:

Parallel.ForEach(listSkills,
    new ParallelOptions { MaxDegreeOfParallelism = 4 },
    item =>
    {
         // some business logic
         var b = getImageFromUrl(item.Url);
         Stream ms = new MemoryStream(b);
    
         saveImage(ms);
         // more business logic
    });