azure - 如何使用 SAS token 上传 10 Gb 文件

Question

我正在尝试使用 SAS token 将大文件(超过 10Gb)上传到 Azure Blob 存储。我像这样生成 token

    val storageConnectionString = s"DefaultEndpointsProtocol=https;AccountName=${accountName};AccountKey=${accountKey}"
    val storageAccount = CloudStorageAccount.parse(storageConnectionString)
    val client = storageAccount.createCloudBlobClient()

    val container = client.getContainerReference(CONTAINER_NAME)
    val blockBlob = container.getBlockBlobReference(path)
    val policy = new SharedAccessAccountPolicy()
    policy.setPermissionsFromString("racwdlup")
    val date = new Date().getTime();
    val expiryDate = new Date(date + 8640000).getTime()
    policy.setSharedAccessStartTime(new Date(date))
    policy.setSharedAccessExpiryTime(new Date(expiryDate))
    policy.setResourceTypeFromString("sco")
    policy.setServiceFromString("bfqt")
    val token = storageAccount.generateSharedAccessSignature(policy)

然后我尝试了 Put Blob API 并遇到了以下错误

$ curl -X PUT -H 'Content-Type: multipart/form-data' -H 'x-ms-date: 2020-09-04' -H 'x-ms-blob-type: BlockBlob' -F <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="92f4fbfef7afd2a3a2f5f0bcf1e1e4" rel="noreferrer noopener nofollow">[email protected]</a>  https://ACCOUNT.blob.core.windows.net/CONTAINER/10gb.csv\?ss\=bfqt\&sig\=.... -v

< HTTP/1.1 413 The request body is too large and exceeds the maximum permissible limit.
< Content-Length: 290
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: f08a1473-301e-006a-4423-837a27000000
< x-ms-version: 2019-02-02
< x-ms-error-code: RequestBodyTooLarge
< Date: Sat, 05 Sep 2020 01:24:35 GMT
* HTTP error before end of send, stop sending
< 
<?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit.
RequestId:f08a1473-301e-006a-4423-837a27000000
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
Time:2020-09-05T01:24:35.7712576Z</Message><MaxLimit>268435456</MaxLimit></Error>%      

之后尝试使用 PageBlob 上传它(我在文档中看到类似大小最多可达 8 TiB 的内容)

$ curl -X PUT -H 'Content-Type: multipart/form-data' -H 'x-ms-date: 2020-09-04' -H 'x-ms-blob-type: PageBlob' -H 'x-ms-blob-content-length: 1099511627776' -F <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="16707f7a732b562726717438756560" rel="noreferrer noopener nofollow">[email protected]</a>  https://ACCOUNT.blob.core.windows.net/CONTAINER/10gb.csv\?ss\=bfqt\&sig\=... -v

< HTTP/1.1 400 The value for one of the HTTP headers is not in the correct format.
< Content-Length: 331
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: b00d5c32-101e-0052-3125-83dee7000000
< x-ms-version: 2019-02-02
< x-ms-error-code: InvalidHeaderValue
< Date: Sat, 05 Sep 2020 01:42:24 GMT
* HTTP error before end of send, stop sending
< 
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:b00d5c32-101e-0052-3125-83dee7000000
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
Time:2020-09-05T01:42:24.5137237Z</Message><HeaderName>Content-Length</HeaderName><HeaderValue>10114368132</HeaderValue></Error>%   

不确定上传如此大文件的正确方法是什么？

Answer 1

在此处检查不同的 blob 类型:https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs

页 blob 实际上将最大大小限制为 8TB，但它最适合随机读写操作。

另一方面:

Block blobs are optimized for uploading large amounts of data efficiently. Block blobs are comprised of blocks, each of which is identified by a block ID. A block blob can include up to 50,000 blocks.

因此 block blob 是最佳选择，因为它支持的大小可达190.7 TB(预览模式)

现在您需要使用 put block https://learn.microsoft.com/en-us/rest/api/storageservices/put-block上传将形成您的 blob 的 block 。