python - 使用 Boto3 (Python) 向 AWS Glacier 进行高级分段上传？

Question

我需要将更大的文件上传到 AWS Glacier 的值。由于大小的原因，我无法在一个请求中完成此操作，因此我需要将其拆分为更小的部分并使用分段上传。

Boto3 是否包含一种方法，可以获取一个大文件，将其分割并逐个上传它的各个部分？正如我所看到的，Java 客户端有这样的方法(取自 AWS 文档):

The high-level API provides a method that you can use to upload archives of any size. Depending on the file you are uploading, the method either uploads an archive in a single operation or uses the multipart upload support in Amazon Glacier to upload the archive in parts.

我尝试过使用:

boto3.client(...).upload_archive(...)

但它不会分割文件，并且我收到错误。

是否有其他方法可以在不实现所有这些低级内容(分割文件、发送多个请求、完成上传...)的情况下实现此目的？

Answer 1

也许这就是您正在寻找的东西？ https://boto3.readthedocs.io/en/latest/reference/services/glacier.html#Glacier.Client.upload_multipart_part

我将从文档中添加一些相关信息。

upload_multipart_part(**kwargs)

This operation uploads a part of an archive. You can upload archive parts in any order. You can also upload them in parallel. You can upload up to 10,000 parts for a multipart upload.

Amazon Glacier rejects your upload part request if any of the following conditions is true:

• SHA256 tree hash does not match To ensure that part data is not corrupted in transmission, you compute a SHA256 tree hash of the part and include it in your request. Upon receiving the part data, Amazon Glacier also computes a SHA256 tree hash. If these hash values don't match, the operation fails.
• Part size does not match The size of each part except the last must match the size specified in the corresponding InitiateMultipartUpload request. The size of the last part must be the same size as, or smaller than, the specified size.
• Range does not align The byte range value in the request does not align with the part size specified in the corresponding initiate request. For example, if you specify a part size of 4194304 bytes (4 MB), then 0 to 4194303 bytes (4 MB - 1) and 4194304 (4 MB) to 8388607 (8 MB - 1) are valid part ranges. However, if you set a range value of 2 MB to 6 MB, the range does not align with the part size and the upload will fail.

这是请求语法

response = client.upload_multipart_part(
vaultName='string',
uploadId='string',
range='string',
body=b'bytes'|file

)

以及参数说明

accountId(字符串)

AccountId 值是拥有保管库的账户的 AWS 账户 ID。您可以指定 AWS 账户 ID，也可以选择指定单个“-”(连字符)，在这种情况下，Amazon Glacier 使用与用于签署请求的凭证关联的 AWS 账户 ID。如果您使用帐户 ID，请勿在 ID 中包含任何连字符(“-”)。

保管库名称(字符串)[必填]

保管库的名称。

uploadId(字符串)[必填]

分段上传的上传ID。

校验和(字符串)

正在上传的数据的 SHA256 树哈希。

范围(字符串)

标识将在此部分中上传的组装存档中的字节范围。 Amazon Glacier 使用此信息以正确的顺序组装存档。此 header 的格式遵循 RFC 2616。 header 示例为 Content-Range:bytes 0-4194303/*。

正文(字节或可查找的类似文件对象)

要上传的数据。