json - 如何使用Hive sQL将ngrams数组字符串和测试频率作为hive表中的单独元素获取？

Question

我正在分析自己的推文，并已使用Hive JSON SerDE 将数据插入到Hive表中。我想在我的推文中找到所有两个单词短语的出现频率，作为表格。输出应类似于:

phrase             frequency
["the","room"]      1248.0
["a","boy"]        1039.0
["rt","to"]        1032.0
["to","ct"]         986.0

现在，我可以对所有单个单词短语执行此操作，并且得到的输出为:

phrase     frequency
["the"]     1248.0
["a"]       1039.0
["rt"]      1032.0
["to"]      986.0
["you"]     828.0

对于单字词组输出，我的代码是:

create table ng(new_ar array<struct<ngram:array<string>,estfrequency:double>>);

INSERT OVERWRITE TABLE ng 
SELECT context_ngrams(sentences(lower(text)),array(null),100) as word 
FROM tweets;

create table wordFreq (ngram array<string>,  estfrequency double);

INSERT OVERWRITE TABLE wordFreq 
SELECT X.ngram, X.estfrequency 
FROM ng LATERAL VIEW explode(new_ar) Z as X;    

select * from wordFreq;

如何为所需的输出修改以上代码？

Answer 1

要将您的代码从1克更改为2克，请将array(null)更改为array(null,null)。