azure - 从 Azure Datalake V2 下载多个文件时出现 System.Net.Http.HttpRequestException

Question

我正在从 Azure Datalake V2 下载超过 1000 个的大量文件，并且不断出现异常:

The SSL connection could not be established, see inner exception. 
<--- Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. 
<--- An existing connection was forcibly closed by the remote host.

堆栈跟踪:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

代码:

var downloadTasks = job.Files.AsParallel().Select(x => Download(x));
await Task.WhenAll(downloadTasks);

private async Task Download(DownloadableFile file)
{
    try
    {
        var options = new BlobRequestOptions
        {
            ParallelOperationThreadCount = 8,
            DisableContentMD5Validation = true,
            StoreBlobContentMD5 = false
        };
        var xzBlob = await _cloudBlobFileService.GetBlockBlobReference(file.FilePath);
        await xzBlob.DownloadToFileAsync(file.LocalFilePath, FileMode.Create, null, options, null);
    }
    catch (Exception e)
    {
         _log.LogCritical(e, "Error downloading " + file.FilePath);
    }
}

我还添加了这个:

ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
ServicePointManager.Expect100Continue = false;

使用.Net core 3.1和WindowsAzure.Storage 9.3.3

到webjob中的program.cs主要方法

我们曾经在没有datalake的情况下配置了blobstorage，但是在切换到datalake之后就出现了这种情况。它不会对应用程序产生太大影响，因为跳过的下载稍后会重试。然而，很高兴知道是什么原因造成的。

Answer 1

您可以先尝试 new storage SDK尽管我不能保证这会解决问题，但该问题已于 11 月上市。这是完全重写

虽然不可能仅从错误消息中准确定位，但有几件事需要注意:

1. 网络错误。这是迄今为止最可能的原因，但有趣的是它与旧的 blob 存储帐户一致地工作。增加超时可能会降低网络错误的频率，重试逻辑将有助于克服这些错误。
2. Using unbounded parallelism不推荐。 ParallelOperationThreadCount 适用于 uploads not downloads ，因此在这种情况下它不会限制请求。默认限制为 server-side connections in .NET is 10 ，推荐increase this使用 .NET Core 时，这是需要考虑的事情。如果您同时访问同一个 blob 或分区太多次，您可能会开始遇到 concurrent connections limits在存储中。