azure - 有关根据数据量对目录进行均匀排序的 Powershell 命令或脚本的指南

Question

我正在将文件服务器从本地迁移到 Azure 文件共享/VM。

我已经准备好一切并准备好移动，但我想将我的文件分成 4 个大小大致相等的批处理。

**问题**

如果我的文件服务器上有 100 个充满数据的目录，名为 Dir1 - Dir100。

什么命令可以帮助我找出如下所示的内容:

Dir_1 – Dir_30 == 数据总量的 25%

Dir_31 -- Dir_65 == 数据总量的 25%

Dir_66 – Dir_90 == 数据总量的25%

Dir_91 – Dir_100 == 数据总量的25%

这有意义吗？

我知道如何获取总数据大小或文件数量......但我无法弄清楚我想要做的事情是否可能或如何做。我一直在胡闹，但我还没有接近。

Answer 1

好吧，所以我两种方式都做了。您没有发布自己的任何内容，因此您必须改编我在这里发布的内容。

首先，创建目录列表。这里的两个分区字段只是它们所属的组。您将需要根据您的评论进行顺序分区。

# Set the number of partitions you need to divide into
$PartitonCount = 4

# I'm making my own list.
# DirectoryName is the FullName of the directory, while Size is the total size of the folder
$Directories = @'
DirectoryName,Size,SequentialPartition,BalancedPartition
Dir001,667117278790,,
Dir002,292429698039,,
Dir003,886665781748,,
Dir004,49665832174,,
Dir005,34041573768,,
Dir006,320236552339,,
Dir007,747674470078,,
Dir008,375284137393,,
Dir009,549754879999,,
Dir010,327528841615,,
Dir011,1079085662940,,
Dir012,1051279115201,,
Dir013,198772106622,,
Dir014,124437323951,,
Dir015,342261556929,,
Dir016,844330660560,,
Dir017,888294129196,,
Dir018,774656795794,,
Dir019,360019686543,,
Dir020,412884330229,,
'@ | ConvertFrom-Csv

现在，对于顺序分区:

# Determine the total size of all directories
$TotalSize = $Directories | Measure-Object -Property Size -Sum | Select-Object -ExpandProperty Sum

# This is the threshold for sequential balancing
$PartitionSizeThreshold = $TotalSize / $PartitonCount

# Initialize the partition size and partition ID
[int64]$CurrentPartitionSize = 0
$CurrentPartition = 0
foreach ($D in $Directories) {
    # Assign the file to the current partition
    $D.SequentialPartition = $CurrentPartition

    # Add the current file's size to the current partition's size
    $CurrentPartitionSize += $D.Size

    # If the current partition's size is over the threshold, go to the next empty partition
    if ($CurrentPartitionSize -gt $PartitionSizeThreshold) {
        $CurrentPartition++
        $CurrentPartitionSize = 0
    }
}

# Results
$Directories

您可以通过以下方式看到它的平衡程度:

# Here is the breakdown of the sequential partitions as a fraction of the total
$Directories |
    Group-Object -Property SequentialPartition |
    ForEach-Object {
         ($_.Group | Measure-Object -Property Size -Sum | Select-Object -ExpandProperty Sum) / $TotalSize 
    }

最后一个分区可能不太平衡。

<小时/>

另一方面，如果需要尽可能平衡:

# Create the partiton size array with a zero for each partition size
$BalancedPartitionSizes = 1..$PartitonCount | ForEach-Object { 0 }

# Sort largest to smallest to assign the largest directories first
$Directories | Sort-Object -Property Size -Descending |
    ForEach-Object {
        # Determine which index of the array has is the smallest size
        $SmallestPartition = $BalancedPartitionSizes | Sort-Object | Select-Object -First 1 | ForEach-Object { [Array]::IndexOf($BalancedPartitionSizes, $_) }

        # Add the directory to the smallest partition
        $BalancedPartitionSizes[$SmallestPartition] += $_.Size
        $_.BalancedPartition = $SmallestPartition
    }

# Results
$Directories | Sort-Object -Property BalancedPartition

您可以看到它如何有效地平衡事物:

# Here is the breakdown of the balanced partitions as a fraction of the total
$Directories |
    Group-Object -Property BalancedPartition |
    ForEach-Object {
        ($_.Group | Measure-Object -Property Size -Sum | Select-Object -ExpandProperty Sum) / $TotalSize 
    }

这更有可能为您提供更平等的平衡，尽管它肯定不完美。