azure - 为什么 Azcopy 适用于 Azure 中的小文件而不适用于大文件？

Question

我可以使用 Azcopy(复制)命令复制文件夹或小文件，但如果我将该命令应用于大文件，传输将失败且没有清晰的日志。

命令:

azcopy copy 'test' 'https://mystorage.blob.core.windows.net/...pKmRpGCVO8B' --recursive --preserve-smb-info=true

Test 是一个包含大文件的文件夹。 (约 10 GB)

当进程被终止时，日志就会停止。输出如下:

INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job a50e4368-5fd0-534b-657d-70216b0e2 has started
Log file is located at: /root/.azcopy/a50e4368-5fd0-534b-657d-70216b0e2.log

Killed

日志仅显示:

cat /root/.azcopy/a50e4368-5fd0-534b-657d-70216b0e2.log
2023/05/23 17:51:19 AzcopyVersion  10.18.1
2023/05/23 17:51:19 OS-Environment  linux
2023/05/23 17:51:19 OS-Architecture  amd64
2023/05/23 17:51:19 Log times are in UTC. Local time is 23 May 2023 17:51:19
2023/05/23 17:51:20 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2023-05-23T17:51:14Z or --include-after=2023-05-23T17:51:14Z
2023/05/23 17:51:20 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2023/05/23 17:51:20 Job-Command copy test https://[...]rw&sr=c&st=2023-05-23t15%3A56%3A08z&sv=2022-11-02 --recursive --preserve-smb-info=true
2023/05/23 17:51:20 Number of CPUs: 1
2023/05/23 17:51:20 Max file buffer RAM 0.500 GB
2023/05/23 17:51:20 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2023/05/23 17:51:20 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2023/05/23 17:51:20 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2023/05/23 17:51:20 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2023/05/23 17:51:20 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2023/05/23 17:51:20 Max open files when downloading: 1048152 (auto-computed)
2023/05/23 17:51:20 Final job part has been created
2023/05/23 17:51:20 JobID=a50e4368-5fd0-534b-657d-70216b4960e2, credential type: Anonymous
2023/05/23 17:51:20 Final job part has been scheduled
2023/05/23 17:51:20 INFO: [P#0-T#0] Starting transfer: Source "/home/user/test/myfile.gz" Destination "https://mystorage.blob.core.windows.net/[...]02". Specified chunk size 8388608

该过程适用于较小的文件，但不适用于大文件。我在日志中找不到 Azcopy 设置的文件限制或相关限制。

Answer 1

您可以通过调整以下值/参数来解决:

--block-size-mb(行内参数):设置卡盘尺寸和

-- AZCOPY_CONCURRENCY_VALUE( session 变量):设置所需的并行化程度(基于可用的空闲内存和 block 大小mb)