json - 如何从文件中传输 JSON？

Question

我将有一个可能非常大的 JSON 文件，我想从中进行流式传输，而不是将其全部加载到内存中。基于JSON::XS中的以下陈述(我添加了强调) ，我相信它不能满足我的需求。是否有 Perl 5 JSON 模块可以从磁盘传输结果？

In some cases, there is the need for incremental parsing of JSON texts. While this module always has to keep both JSON text and resulting Perl data structure in memory at one time, it does allow you to parse a JSON stream incrementally. It does so by accumulating text until it has a full JSON object, which it then can decode. This process is similar to using decode_prefix to see if a full JSON object is available, but is much more efficient (and can be implemented with a minimum of method calls).

澄清一下，JSON 将包含一个对象数组。我想一次从文件中读取一个对象。

Answer 1

在易用性和速度方面，JSON::SL似乎是赢家:

#!/usr/bin/perl

use strict;
use warnings;

use JSON::SL;

my $p = JSON::SL->new;

#look for everthing past the first level (i.e. everything in the array)
$p->set_jsonpointer(["/^"]);

local $/ = \5; #read only 5 bytes at a time
while (my $buf = <DATA>) {
    $p->feed($buf); #parse what you can
    #fetch anything that completed the parse and matches the JSON Pointer
    while (my $obj = $p->fetch) {
        print "$obj->{Value}{n}: $obj->{Value}{s}\n";
    }
}

__DATA__
[
    { "n": 0, "s": "zero" },
    { "n": 1, "s": "one"  },
    { "n": 2, "s": "two"  }
]

JSON::Streaming::Reader还可以，但它速度较慢，并且界面过于冗长(所有这些代码引用都是必需的，即使许多代码引用什么都不做):

#!/usr/bin/perl

use strict;
use warnings;

use JSON::Streaming::Reader;

my $p = JSON::Streaming::Reader->for_stream(\*DATA);

my $obj;
my $attr;
$p->process_tokens(
    start_array    => sub {}, #who cares?
    end_array      => sub {}, #who cares?
    end_property   => sub {}, #who cares?
    start_object   => sub { $obj = {}; },     #clear the current object
    start_property => sub { $attr = shift; }, #get the name of the attribute
    #add the value of the attribute to the object
    add_string     => sub { $obj->{$attr} = shift; },
    add_number     => sub { $obj->{$attr} = shift; },
    #object has finished parsing, it can be used now
    end_object     => sub { print "$obj->{n}: $obj->{s}\n"; },
);

__DATA__
[
    { "n": 0, "s": "zero" },
    { "n": 1, "s": "one"  },
    { "n": 2, "s": "two"  }
]

解析 1,000 条记录需要 JSON::SL 0.2 秒，JSON::Streaming::Reader 3.6 秒(注意，JSON::SL 一次输入 4k，我无法控制 JSON::Streaming::Reader 的缓冲区大小)。