hadoop - 使用Pig将最大值填充到同一键的相邻记录

Question

我在下面设置了数据

key,value
---------
key1|10
key1|20
key1|30
key2|50
key2|70

我需要使用最大“值”列填充同一键的新列。

输出必须是

key1|10|30
key1|20|30
key1|30|30
key2|50|70
key2|70|70

Below is the Pig script, but facing issues.
A = LOAD 'input.txt' using PigStorage('|');
B = foreach A generate $0,$1,min($1); 


grunt> A = LOAD 'input.txt' using PigStorage('|');
grunt> B = foreach A generate $0,$1,max($1);

2017-05-26 06:48:02,347 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve max using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]

Answer 1

下面的代码应该做。请记住，在使用group，MAX和MIN之类的功能之前，需要首先对关系进行AVG。

A = load 'file' using PigStorage(',') as (id: chararray, val: int);
B = GROUP A by id;
C = FOREACH B GENERATE FLATTEN(group), MAX(A.val) as (maxval: int);
D = JOIN A by id, C BY group;
E = FOREACH D generate A::id, A::val, C::maxval;
DUMP E;

运行此命令，您应该获得:

(key1,30,30)
(key1,20,30)
(key1,10,30)
(key2,70,70)
(key2,50,70)