asp.net - 从 ASP.NET Core Web API 将多 GB 文件流式传输到 AWS S3

Question

我希望通过 ASP.NET Core Web API 在 AWS S3 存储桶中创建一个大型(多 GB)文件。该文件足够大，我不希望在将 Stream 上传到 AWS S3 之前将其加载到内存中。

使用 PutObjectAsync() 我被迫在将 Stream 传递到 AWS SDK 之前预先填充它，如下所示:

var putObjectRequest = new PutObjectRequest
{
    BucketName = "my-s3-bucket",
    Key = "my-file-name.txt",
    InputStream = stream
};
var putObjectResponse = await amazonS3Client.PutObjectAsync(putObjectRequest);

我的理想模式是 AWS SDK 返回一个 StreamWriter(某种)，我可以 Write() 多次，然后 Finalise()当我完成时。

有关我的挑战的两个问题:

• 我是否误解了在调用 PutObjectAsync() 之前必须预先填充 Stream？
• 我应该如何上传大型(多 GB)文件？

Answer 1

对于这种情况AWS docs提供两个选项:

• Using the AWS .NET SDK for Multipart Upload (High-Level API)
• Using the AWS .NET SDK for Multipart Upload (Low-Level API)

高级 API 只是建议您创建一个 TransferUtilityUploadRequest指定了 PartSize，因此类本身可以上传文件，而无需自己维护上传。在这种情况下，您可以通过订阅 StreamTransferProgress 获取分段上传的进度。事件。您可以上传文件、流或目录。

Low-level API显然，更复杂，但更灵活 - 您可以启动上传，然后循环上传文件的下一部分。文档中的示例代码:

var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);

// List to store upload part responses.
var uploadResponses = new List<UploadPartResponse>();

// 1. Initialize.
var initiateRequest = new InitiateMultipartUploadRequest
    {
        BucketName = existingBucketName,
        Key = keyName
    };

var initResponse = s3Client.InitiateMultipartUpload(initRequest);

// 2. Upload Parts.
var contentLength = new FileInfo(filePath).Length;
var partSize = 5242880; // 5 MB

try
{
    long filePosition = 0;
    for (var i = 1; filePosition < contentLength; ++i)
    {
        // Create request to upload a part.
        var uploadRequest = new UploadPartRequest
            {
                BucketName = existingBucketName,
                Key = keyName,
                UploadId = initResponse.UploadId,
                PartNumber = i,
                PartSize = partSize,
                FilePosition = filePosition,
                FilePath = filePath
            };

       // Upload part and add response to our list.
       uploadResponses.Add(s3Client.UploadPart(uploadRequest));

       filePosition += partSize;
   }

   // Step 3: complete.
   var completeRequest = new CompleteMultipartUploadRequest
       {
           BucketName = existingBucketName,
           Key = keyName,
           UploadId = initResponse.UploadId,
        };

   // add ETags for uploaded files
   completeRequest.AddPartETags(uploadResponses);

   var completeUploadResponse = s3Client.CompleteMultipartUpload(completeRequest);     
}
catch (Exception exception)
{
    Console.WriteLine("Exception occurred: {0}", exception.ToString());
    var abortMPURequest = new AbortMultipartUploadRequest
        {
            BucketName = existingBucketName,
            Key = keyName,
            UploadId = initResponse.UploadId
        };
    s3Client.AbortMultipartUpload(abortMPURequest);
}

Asynchronous version of UploadPart也可用，因此如果您需要完全控制上传，则应该调查该路径。