hadoop - 如何从表中获取开始和结束事件

Question

我在一个表中有如下记录

session_id sequence timestamp
1           1       298349
1           2       299234
1           3       234255
2           1       153523
2           2       234524
3           1       123434 

我想得到如下结果

session_id  start       end
1           298349      234255
2           153523      234524
3           123434      123434

我怎样才能在 pig 身上做到这一点？

Answer 1

register 'file:$piglib/datafu-1.2.0.jar';

define FirstTupleFromBag datafu.pig.bags.FirstTupleFromBag();

input_data = load 'so.txt' using PigStorage('\t') as (session_id:int, sequence:int, time:long);

g = group input_data by session_id;

r = foreach g {
    s1 = order input_data by sequence asc;
    s2 = order input_data by sequence desc;
    generate group as session_id, FirstTupleFromBag(s1, null).time as start, FirstTupleFromBag(s2, null).time as end;
}

dump r;

首先按session_id分组，然后按序列升序和降序排序，分别取排序后包的第一个元组。

这利用了 datafu UDF 库 ( http://datafu.incubator.apache.org/docs/datafu/1.2.0/datafu/pig/bags/FirstTupleFromBag.html )