powershell - 从 | 中删除 3,7 和 9 列的速度很慢使用 PowerShell 分离 txt 文件

Question

我有包含大量数据的管道分隔数据文件，我想删除 3、7 和 9 列。下面的脚本工作 100% 正常。但是 22MB 的文件需要 5 分钟，速度太慢了。

Adeel|01|test|1234589|date|amount|00|123345678890|test|all|01|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|05|test|1234589|date|amount|00|123345678890|test|all|05|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|09|test|1234589|date|amount|00|123345678890|test|all|09|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00|Adeel|12|test|1234589|date|amount|00|123345678890|test|all|12|

    param
(
    # Input data file
    [string]$Path = 'O:\Temp\test.txt',
    # Columns to be removed, any order, dupes are allowed
    [int[]]$Remove = (3,6)
)

# sort indexes descending and remove dupes
$Remove = $Remove | Sort-Object -Unique -Descending

# read input lines
Get-Content $Path | .{process{
    # split and add to ArrayList which allows to remove items
    $list = [Collections.ArrayList]($_ -split '\|')

    # remove data at the indexes (from tail to head due to descending order)
    foreach($i in $Remove) {
        $list.RemoveAt($i)
    }

    # join and output
    #$list -join '|'
    $contentUpdate=$list -join '|'
    Add-Content "O:\Temp\testoutput.txt" $contentUpdate
}
}

Answer 1

Get-Content 相对较慢。使用管道会增加额外的开销。

当性能很重要时，StreamReader和 StreamWriter可能是更好的选择:

param (
    # Input data file
    [string] $InputPath = 'input.txt',
    # Output data file
    [string] $OutputPath = 'output.txt',
    # Columns to be removed, any order, dupes are allowed
    [int[]] $Remove = (1, 2, 2),
    # Column separator
    [string] $Separator = '|',
    # Input file encoding
    [Text.Encoding] $Encoding = [Text.Encoding]::Default
)

$ErrorActionPreference = 'Stop'

# Gets rid of dupes and provides fast lookup ability
$removeSet = [Collections.Generic.HashSet[int]] $Remove

$reader = $writer = $null

try {
    $reader = [IO.StreamReader]::new(( Convert-Path -LiteralPath $InputPath ), $encoding )

    $null = New-Item $OutputPath -ItemType File -Force  # as Convert-Path requires existing path

    while( $line = $reader.ReadLine() ) {

        if( -not $writer ) {
            # Construct writer only after first line has been read, so $reader.CurrentEncoding is available 
            $writer = [IO.StreamWriter]::new(( Convert-Path -LiteralPath $OutputPath ), $false, $reader.CurrentEncoding )
        }

        $columns = $line.Split( $separator )
        $isAppend = $false

        for( $i = 0; $i -lt $columns.Length; $i++ ) {
            if( -not $removeSet.Contains( $i ) ) {
                if( $isAppend ) { $writer.Write( $separator ) }
                $writer.Write( $columns[ $i ] )
                $isAppend = $true
            }
        }

        $writer.WriteLine()  # Write (CR)LF
    }
}
finally {
    # Make sure to dispose the reader and writer so files get closed.
    if( $writer ) { $writer.Dispose() }
    if( $reader ) { $reader.Dispose() }
}

使用
• Convert-Path 是因为 .NET 与 PowerShell 有不同的当前目录，因此最好将绝对路径传递给 .NET API。
• 如果这仍然不够快，请考虑改用 C# 编写。特别是对于这样的“低级”代码，C# 往往更快。您可以使用 Add-Type -TypeDefinition $csCode 在 PowerShell 中嵌入 C# 代码。
• 作为另一个优化，您可以使用 String.IndexOf() 和 ，而不是使用创建比实际需要更多的子字符串的 String.Split() String.Substring() 以仅提取必要的列。
• 最后一点，您可以尝试使用 StreamReader和 StreamWriter constructors允许您指定缓冲区大小。