c# - parallel.foreach 和 httpclient - 奇怪的行为

Question

我有一段代码循环遍历集合并为每次迭代调用 httpclient。 httpclient 调用的 api 平均需要 30-40ms 来执行。按顺序调用它，我得到了预期的结果，但是一旦我使用 Parallel.foreach，它就会花费更长的时间。仔细查看日志，我可以看到相当多的 httpclient 调用需要更多的 1000 毫秒才能执行，然后时间又回落到 30-40 毫秒。查看 api 日志，我可以看到它几乎不会超过 100 毫秒。我不确定为什么会出现这种尖峰。

代码是

using (var client = new HttpClient())
{
  var content = new StringContent(parameters, Encoding.UTF8, "application/json");
  var response = client.PostAsync(url, content);
  _log.Info(string.Format("Took {0} ms to send post", watch.ElapsedMilliseconds));
  watch.Restart();

  var responseString = response.Result.Content.ReadAsStringAsync();
  _log.Info(string.Format("Took {0} ms to readstring after post", watch.ElapsedMilliseconds));
}

并行调用是这样的

    Console.WriteLine("starting parallel...");
    Parallel.ForEach(recipientCollections, recipientCollection => 
      {    
        // A lot of processing happens here to create relevant content
        var secondaryCountryRecipientList = string.Join(",",refinedCountryRecipients);
        var emailApiParams = new SendEmailParametersModel(CountrySubscriberApplicationId,
                                        queueItem.SitecoreId, queueItem.Version, queueItem.Language, countryFeedItem.Subject,
                                        countryFeedItem.Html, countryFeedItem.From, _recipientsFormatter.Format(secondaryCountryRecipientList));

       log.Info(string.Format("Sending email request for {0}. Recipients {1}",                                        queueItem.SitecoreId, secondaryCountryRecipientList));

        var response = _notificationsApi.Invoke(emailApiParams);
        });

谢谢

Answer 1

默认情况下，.NET 只允许每个服务器有 2 个连接。要更改此设置，您必须更改 ServicePointManager.DefaultConnectionLimit 的值到更大的值，例如 20 或 100。

如果您发出太多请求，这不会阻止服务器泛滥或消耗过多内存。更好的选择是使用 ActionBlock< T>缓冲请求并在受控函数中并行发送它们，例如:

 ServicePointManager.DefaultConnectionLimit =20;

 var client = new HttpClient();

 var blockOptions=new ExecutionDataflowBlockOptions{MaxDegreeOfParallelism=10};

 var emailBlock=new ActionBlock<SendEmailParametersModel>(async arameters=>
     {
         var watch=new Stopwatch();
         var content = new StringContent(parameters, Encoding.UTF8, "application/json");
         var response = await client.PostAsync(url, content);
         _log.Info(..);
         watch.Restart();

         var responseString = await response.Result.Content.ReadAsStringAsync();
         _log.Info(...);
 });

发送电子邮件不再需要并行调用:

foreach(var recipientCollection in recipientCollections)
{
    var secondaryCountryRecipientList = string.Join(",",refinedCountryRecipients);
    var emailApiParams = new SendEmailParametersModel(CountrySubscriberApplicationId, queueItem.SitecoreId, queueItem.Version, queueItem.Language, countryFeedItem.Subject,countryFeedItem.Html, countryFeedItem.From, _recipientsFormatter.Format(secondaryCountryRecipientList));

   emailBlock.Post(emailApiParams);
   log.Info(...);
}
emailBlock.Complete();
await emailBlock.Completion();

HttpClient 是线程安全的，允许您对所有请求使用相同的客户端。

上面的代码将缓冲所有请求并一次执行 10 个。调用 Complete() 告诉 block 完成所有事情并停止处理新消息。 await emailBlock.Completion() 在继续之前等待所有现有消息完成