perl - 如何在perl编程中快速读取.gz文件中的数据

Question

我正在阅读一个大约 3 GB 的 .gz 文件。我正在使用 Perl 程序 grepping 一个模式。我能够 grep 模式，但处理时间太长。谁能帮助我如何快速处理？

use strict ;
use warnings ;
use Compress::Zlib;
my $file = "test.gz";
my $gz = gzopen ($file, "rb") or die "Error Reading $file: $gzerrno";
while ($gz->gzreadline($_) > 0 ) {
if (/pattern/) {
print "$_----->PASS\n";
}
}
die "Error reading $file: $gzerrno" if $gzerrno != Z_STREAM_END;
$gz ->gzclose();

Z_STREAM_END 变量有什么作用？

Answer 1

我写了一个脚本来记录各种方法读取 gz 文件所花费的时间。我也发现 Compress::Zlib 非常慢。

use strict;
use warnings;
use autodie ':all';
use Compress::Zlib;
use Time::HiRes 'time';

my $file = '/home/con/Documents/snp150.txt.gz';
# time zcat execution
my $start_zcat = Time::HiRes::time();
open my $zcat, "zcat $file |";
while (<$zcat>) {
  #      print $_;
}
close $zcat;
my $end_zcat = Time::HiRes::time();
# time Compress::Zlib reading
my $start_zlib = Time::HiRes::time();
my $gz = gzopen($file, 'r') or die "Error reading $file: $gzerrno";
while ($gz->gzreadline($_) > 0) {#http://blog-en.openalfa.com/how-to-read-and-write-compressed-files-in-perl
#       print "$_";# Process the line read in $_
}
$gz->gzclose();
my $end_zlib = Time::HiRes::time();

printf("zlib took %lf seconds.\n", $end_zlib - $start_zlib);
printf("zcat took %lf seconds.\n", $end_zcat - $start_zcat);

使用这个脚本，我发现通过 zcat 读取比 Compress::Zlib 快 7 倍(!)当然是文件。