hadoop - 对 Pig 中袋子的不同值进行计数

Question

在执行看似两级分组时，我对 Pig 有疑问。例如，假设我有一些示例输入数据，例如:

email_id:chararray    from:chararray        to:bag{recipients:tuple(recipient:chararray)}
e1                    user1@example.com     {(friend1@example.com),(friend2@example.com),(friend3@myusers.com)}
e2                    user1@example.com     {(friend1@example.com),(friend4@example.com)}
e3                    user1@example.com     {(friend5@example.com)}
e4                    user2@example.com     {(friend2@example.com),(friend4@example.com)}

所以每一行都是一封从用户“发件人”到用户“收件人”的电子邮件。

我最终想要一个所有发件人和他们向其发送电子邮件的所有人的列表，包括每个人发送的电子邮件的数量，从高到低排序，例如:

user1@example.com     {(friend1@example.com, 2), (friend2@example.com, 1), (friend3@example.com, 1), (friend4@example.com, 1), (friend5@example.com, 1)}
user2@example.com     {(friend2@example.com, 1), (friend4@example.com, 1)}

如果能提出在 Pig 中解决此问题的最佳方法，我们将不胜感激!

Answer 1

这是脚本的一个版本:

inpt = load '/pig_data/pig_fun/input/from_senders.txt' as (email_id:chararray, from:chararray, to:bag{recipients:tuple(recipient:chararray)});

pivot = foreach inpt generate from, FLATTEN(to);
pivot = foreach pivot generate from, to::recipient as recipient;
dump pivot;
/*
(user1@example.com,friend1@example.com)
(user1@example.com,friend2@example.com)
(user1@example.com,friend3@myusers.com)
(user1@example.com,friend1@example.com)
(user1@example.com,friend4@example.com)
(user1@example.com,friend5@example.com)
(user2@example.com,friend2@example.com)
(user2@example.com,friend4@example.com)
*/

grp = group pivot by (from, recipient);
with_count = foreach grp generate FLATTEN(group), COUNT(pivot) as count;
dump with_count;
/*
(user1@example.com,friend1@example.com,2)
(user1@example.com,friend2@example.com,1)
(user1@example.com,friend3@myusers.com,1)
(user1@example.com,friend4@example.com,1)
(user1@example.com,friend5@example.com,1)
(user2@example.com,friend2@example.com,1)
(user2@example.com,friend4@example.com,1)
*/

to_bag = group with_count by from;
result = foreach to_bag {
    order_by_count = order with_count by count desc;
    generate group as from, order_by_count.(recipient, count);
};
dump result;
/*
(user1@example.com,{(friend1@example.com,2),(friend2@example.com,1),(friend3@myusers.com,1),(friend4@example.com,1),(friend5@example.com,1)})
(user2@example.com,{(friend2@example.com,1),(friend4@example.com,1)})
*/

希望对您有所帮助。