hadoop - pig : Pivoting & Sum 3 relations

Question

我有 3 种不同的关系，如下所述，我可以使用 UDF 获取输出，但在 PIG 中寻找实现。在论坛中提到了其他东西，但没有对这个问题有具体的想法。

过程:

FN1,10
FN2,20
FN3,23
FN4,25
FN5,15
FN7,40
FN10,56

拒绝:

FN1,12
FN2,13
FN3,33
FN6,60
FN8,23
FN9,44
FN10,4

所有FN:

FN1
FN2
FN3
FN4
FN5
FN6
FN7
FN8
FN9
FN10

所需的输出是:

FN1,10,12,22
FN2,20,13,33
FN3,23,33,56
FN4,25,0,25
FN5,15,0,15
FN6,0,60,60
FN7,40,0,40
FN8,0,23,23
FN9,0,44,44
FN10,56,4,60

Answer 1

假设你的关系在 test.txt test2.txt 和 test3.txt

A = LOAD 'test.txt' using PigStorage(',');
B = LOAD 'test2.txt' using PigStorage(',');
C = LOAD 'test3.txt' using PigStorage(',');
D = COGROUP A by $0, B by $0;
E = COGROUP C by $0, D by $0;
F = FOREACH E generate $0, FLATTEN(D.A), FLATTEN(D.B);
G = FOREACH F generate $0, $1.$1, $2.$1;
H = FOREACH G generate $0, FLATTEN((IsEmpty($1)?null:$1)), FLATTEN((IsEmpty($2)?null:$2));
I = foreach H generate $0, ($1 is null?0:$1),($2 is null?0:$2),($1 is null?0:$1)+($2 is null?$0:$2);
dump I;

输出

(FN1,10,12,22)
(FN2,20,13,33)
(FN3,23,33,56)
(FN4,25,0,)
(FN5,15,0,)
(FN6,0,60,60)
(FN7,40,0,)
(FN8,0,23,23)
(FN9,0,44,44)
(FN10,56,4,60)