powershell - powershell 将文件移动到 Amazon s3

Question

我有下面的 PowerShell 脚本，可以为我将文件移动到我的亚马逊存储桶，并且对于一些小文件一切正常，但是当复制较大的文件时，for 循环继续循环并开始他们在其他人完成之前复制并且它没有'很快我就可以同时传输 100 个文件。

我想要的是能够将同时文件传输的数量限制为 5 或 10？

foreach ($line in $csv) {  

#--------------------Transfer files Put in a for each loop here---------------------------
$SourceFolder  =$line.destination
$sourceFile = $line.name

if(test-Path -path $SourceFolder){
    Write-S3Object -BucketName $BucketName  -Key $sourceFile  -File  $SourceFolder 
    #check fro missing files
        $S3GetRequest = New-Object Amazon.S3.Model.S3Object  #get-S3Object  -BucketName   $BucketName  -Key $sourceFile
        $S3GetRequest = get-S3Object  -BucketName $BucketName  -Key $sourceFile

        if($S3GetRequest -eq $null){
            Write-Error "ERROR: Amazon S3 get requrest failed. Script halted."
            $sourceFile + ",Transfer Error" |out-file $log_loc -append
    }
}else {$SourceFolder + ",Missing File Error" |out-file $log_loc -append}

}

Answer 1

从描述来看，您的大文件似乎触发了分段上传。来自Write-S3Object documentation :

If you are uploading large files, Write-S3Object cmdlet will use multipart upload to fulfill the request. If a multipart upload is interrupted, Write-S3Object cmdlet will attempt to abort the multipart upload.

不幸的是，Write-S3Object 并没有真正的原生方式来处理您的用例。然而，Multipart Upload Overview描述了我们可以利用的行为:

Multipart uploading is a three-step process: You initiate the upload, you upload the object parts, and after you have uploaded all the parts, you complete the multipart upload. Upon receiving the complete multipart upload request, Amazon S3 constructs the object from the uploaded parts, and you can then access the object just as you would any other object in your bucket.

这让我怀疑我们可以使用 Get-S3Object ping 我们的对象来查看它们是否存在。如果没有，我们应该等到他们上传更多文件。

我在下面创建了一个脚本来执行此操作——它遍历一组文件并在您上传文件时收集它们的名称。一旦超过 5 个上传文件，脚本将检查它们是否存在，如果存在则继续。否则，它将继续检查它们是否存在。

$BucketName = "myS3Bucket"
$s3Directory = "C:\users\$env:username\documents\s3test"
$concurrentLimit = 5
$inProgressFiles = @()

foreach ($i in Get-ChildItem $s3Directory) 
{ 
  # Write the file to S3 and add the filename to a collection.
  Write-S3Object -BucketName $BucketName -Key $i.Name -File $i.FullName 
  $inProgressFiles += $i.Name

  # Wait to continue iterating through files if there are too many concurrent uploads
  while($inProgressFiles.Count -gt $concurrentLimit) 
  {
    Write-Host "Before: "$($inProgressFiles.Count)

    # Reassign the array by excluding files that have completed the upload to S3.
    $inProgressFiles = @($inProgressFiles | ? { @(get-s3object -BucketName $BucketName -Key $_).Count -eq 0 })

    Write-Host "After: "$($inProgressFiles.Count)

    Start-Sleep -s 1
  }

  Start-Sleep -s 1
}

您可以根据自己的需要修改它，方法是更改​​ foreach 循环以使用您的 csv 内容。我添加了 sleep 语句，以便您能够观看它并了解它是如何工作的——请随意更改/删除它们。