out-of-memory - Azure 函数内存不足异常

Question

我正在从 Blob 读取 PDF 文件并使用 Azure Function 进行一些操作。当 PDF 大小为 1.5GB(或更大)时，我的 azure 函数在代码命中命令下方时立即失败并出现内存不足异常。

var ms = new MemoryStream();
log.LogInformation("Converting this File to memorystream : " + blob.Uri);
blob.DownloadToStream(ms);  //Failes HERE.

我尝试将计划增加到 EP3(14 GB 内存 + 840 ACU)。但问题还是一样。我还需要更改其他一些配置吗？如何处理。

Microsoft.Azure.Storage.StorageException: Exception of type 'System.OutOfMemoryException' was thrown.
 ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.MemoryStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Storage.Core.Util.AsyncStreamCopier`1.StartCopyStreamAsyncHelper(Nullable`1 copyLength, Nullable`1 maxLength, CancellationToken token)
   at Microsoft.Azure.Storage.Core.Util.AsyncStreamCopier`1.StartCopyStreamAsync(Nullable`1 copyLength, Nullable`1 maxLength, CancellationToken cancellationToken)
   at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.Azure.Storage.Core.Executor.Executor.<>c__DisplayClass0_0`1.<ExecuteSync>b__0()
   at Microsoft.Azure.Storage.Core.Util.CommonUtility.RunWithoutSynchronizationContext[T](Func`1 actionToRun)
   at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   at Microsoft.Azure.Storage.Blob.CloudBlob.DownloadRangeToStream(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.Azure.Storage.Blob.CloudBlob.DownloadToStream(Stream target, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)

Answer 1

This blog听起来它已经确定了问题 - MemoryStream 采用 int32 来设置其容量，没有选择使用更大的数字。我无法完全协调 int32.MaxValue (2147483647) 的值与指定大小(1.5GiB 字节为 1610612736 字节)，但它似乎足够接近成为罪魁祸首。该博客提出了一种将大量内容写入 Blob 存储的解决方案，但我看不出这对您的用例有何作用。

如果您确实需要使用 MemoryStream 来满足您正在使用的库的需求，那么处理这么大的文件将是不可能的。