Perl 段落 n 元语法

Question

假设我有一个文本句子:

$body = 'the quick brown fox jumps over the lazy dog';

我想将该句子放入“关键字”的散列中，但我想允许多单词关键字；我有以下方法来获取单个单词关键字:

$words{$_}++ for $body =~ m/(\w+)/g;

完成后，我有一个如下所示的哈希:

'the' => 2,
'quick' => 1,
'brown' => 1,
'fox' => 1,
'jumps' => 1,
'over' => 1,
'lazy' => 1,
'dog' => 1

下一步，以便我可以获得 2 个单词的关键字:

$words{$_}++ for $body =~ m/(\w+ \w+)/g;

但这只能得到所有“其他”对；看起来像这样:

'the quick' => 1,
'brown fox' => 1,
'jumps over' => 1,
'the lazy' => 1

我还需要一个字的偏移量:

'quick brown' => 1,
'fox jumps' => 1,
'over the' => 1

还有比下面更简单的方法吗？

my $orig_body = $body;
# single word keywords
$words{$_}++ for $body =~ m/(\w+)/g;
# double word keywords
$words{$_}++ for $body =~ m/(\w+ \w+)/g;
$body =~ s/^(\w+)//;
$words{$_}++ for $body =~ m/(\w+ \w+)/g;
$body = $orig_body;
# triple word keywords
$words{$_}++ for $body =~ m/(\w+ \w+ \w+)/g;
$body =~ s/^(\w+)//;
$words{$_}++ for $body =~ m/(\w+ \w+ \w+)/g;
$body = $orig_body;
$body =~ s/^(\w+ \w+)//;
$words{$_}++ for $body =~ m/(\w+ \w+ \w+)/g;

Answer 1

虽然所描述的任务对于手动编码可能很有趣，使用处理 n-gram 的现有 CPAN 模块不是更好吗？看起来像 Text::Ngrams (与 Text::Ngram 相对)可以处理基于单词的 n-gram 分析。