regex - 按第二列中的时间戳合并多个日志文件

Question

同一天我有多个日志文件。我想要做的是使用 Perl 脚本根据日志中的时间戳将它们合并为一个。

Log-1.log

2014-06-02 21:54:38,805 INFO  com.HomeManeger [Executor:Thread-19]: MyInfo started for myid=TEST-401406
2014-06-02 21:56:27,358 INFO  com.HomeManeger [Executor:Thread-13]: HomeManeger: populateMyInfo completed for my id = TEST-401406,  
2014-06-02 21:59:32,358 INFO  com.HomeManeger [Executor:Thread-17]: MyInfo completed for myid=TEST-401405

Log-2.log:

2014-06-02 21:56:27,295 INFO  com.homeManeger.MyCommand [Proxy:ProxyService:TcpWorker:2]:     MyCommand::Processing reqest[AB:MyInfo] obj(Collection [ID={005A004A5B0F9}, ]
) client(POFFBObj [ID={XXXXXX-E8F5-11D5-YYY-0002B33D9D0C}, meta={}, fields=[XXX]]
)
2014-06-02 21:58:27,310 INFO  com.HomeManeger.UpdateMyInfoTask

合并日志:

2014-06-02 21:54:38,805 INFO  com.HomeManeger [Executor:Thread-19]: MyInfo started for myid=TEST-401406
2014-06-02 21:56:27,295 INFO  com.homeManeger.MyCommand [Proxy:ProxyService:TcpWorker:2]: MyCommand::Processing reqest[AB:MyInfo] obj(Collection[ID={005A004A5B0F9}, ]
) client(POFFBObj [ID={XXXXXX-E8F5-11D5-YYY-0002B33D9D0C}, meta={}, fields=[XXX]]
)
2014-06-02 21:56:27,358 INFO  com.HomeManeger [Executor:Thread-13]: HomeManeger: populateMyInfo completed for my id = TEST-401406,  
2014-06-02 21:56:32,358 INFO  com.HomeManeger [Executor:Thread-17]: MyInfo completed for myid=TEST-401405    
2014-06-02 21:58:27,310 INFO  com.HomeManeger.UpdateMyInfoTask

我是 Perl 的新手，如有任何帮助，我们将不胜感激。

Answer 1

下面的脚本可以处理任意数量的日志文件，并将合并后的日志输出到一个文件中。

它会将它们全部加载到内存中，因此大小是一个因素:

#!/usr/bin/perl

use warnings;
use strict;

die "Usage: perl $0 log1 log2 > merged.log\n" if !@ARGV;

my @lines;

while (<>) {
    if (/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/) {
        push @lines, $_;
    } else {
        $lines[-1] .= $_;
    }
}

print sort @lines;

如果内存是一个因素，您将需要进行正式的合并排序。

以下改编自perl merge sort从今年 3 月开始:

use strict;
use warnings;
use autodie;

die "Usage: perl $0 log1 log2 > merged.log\n" if !@ARGV;

# Initialize File handles
my @fhs = map {open my $fh, '<', $_; $fh} @ARGV;

# First Line of each file
my @data = map {scalar <$_>} @fhs;

# Loop while a next line exists
while (@data) {
    # Pull out the next entry.
    my $index = (sort {$data[$a] cmp $data[$b]} (0..$#data))[0];

    print $data[$index];

    # Fill In next Data at index.
    while (defined($data[$index] = readline $fhs[$index])) {
        last if $data[$index] =~ /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/;
        print $data[$index];
    }

    # End of that File
    if (! defined($data[$index]) {
        splice @fhs, $index, 1;
        splice @data, $index, 1;
    }
}