amazon-web-services - 使用 MultiPartUpload 复制 S3 对象

Question

我需要重命名 AWS S3 中的大量对象。对于小对象，以下代码片段可以完美运行:

input := &s3.CopyObjectInput{
    Bucket:     aws.String(bucket),
    Key:        aws.String(targetPrefix),
    CopySource: aws.String(source),
}
_, err = svc.CopyObject(input)
if err != nil {
    panic(errors.Wrap(err, "error copying object"))
}

我遇到了较大对象的 S3 大小限制。我知道我需要使用多部分上传来复制对象。到目前为止，这是我尝试过的:

multiPartUpload, err := svc.CreateMultipartUpload(
    &s3.CreateMultipartUploadInput{
        Bucket: aws.String(bucket),
        Key: aws.String(targetPrefix), // targetPrefix is the new name
    },
)
if err != nil {
    panic(errors.Wrap(err, "could not create MultiPartUpload"))
}
resp, err := svc.UploadPartCopy(
    &s3.UploadPartCopyInput{
        UploadId:   multiPartUpload.UploadId,
        Bucket:     aws.String(bucket),
        Key:        aws.String(targetPrefix),
        CopySource: aws.String(source),
        PartNumber: aws.Int64(1),
    },
)
if err != nil {
    panic(errors.Wrap(err, "error copying multipart object"))
}
log.Printf("copied: %v", resp)

golang SDK 帮我解决了问题:

InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120

我也尝试过以下方法，但我没有在这里列出任何部分:

multiPartUpload, err := svc.CreateMultipartUpload(
    &s3.CreateMultipartUploadInput{
        Bucket: aws.String(bucket),
        Key: aws.String(targetPrefix), // targetPrefix is the new name
    },
)
if err != nil {
    panic(errors.Wrap(err, "could not create MultiPartUpload"))
}
err = svc.ListPartsPages(
    &s3.ListPartsInput{
        Bucket:   aws.String(bucket),       // Required
        Key:      obj.Key,                  // Required
        UploadId: multiPartUpload.UploadId, // Required
    },
    // Iterate over all parts in the `CopySource` object
    func(parts *s3.ListPartsOutput, lastPage bool) bool {
        log.Printf("parts:\n%v\n%v", parts, parts.Parts)
        // parts.Parts is an empty slice
        for _, part := range parts.Parts {
            log.Printf("copying %v part %v", source, part.PartNumber)
            resp, err := svc.UploadPartCopy(
                &s3.UploadPartCopyInput{
                    UploadId:   multiPartUpload.UploadId,
                    Bucket:     aws.String(bucket),
                    Key:        aws.String(targetPrefix),
                    CopySource: aws.String(source),
                    PartNumber: part.PartNumber,
                },
            )
            if err != nil {
                panic(errors.Wrap(err, "error copying object"))
            }
            log.Printf("copied: %v", resp)
        }
        return true
    },
)
if err != nil {
    panic(errors.Wrap(err, "something went wrong with ListPartsPages!"))
}

我做错了什么或者我误解了什么？

Answer 1

我认为 ListPartsPages 是错误的方向，因为它适用于“分段上传”，这是不同于 s3“对象”的实体。因此，您将已上传的部分列出到您刚刚创建的分段上传中。

您的第一个示例已接近所需，但您需要手动将原始文件拆分为多个部分，每个部分的范围由 UploadPartCopyInput 的 CopySourceRange 指定.至少这是我阅读文档后的收获。