hadoop - 在 pig 中，如何计算包含特定字符串的行数？

Question

假设我有一组目标词:

a b c d

和一个输入文件:

a d f s g e
12399
c a d i f
a 2

那么我应该返回:

a 3
b 0
c 1
d 2

我怎样才能在 pig 身上做到这一点？谢谢!

Answer 1

首先从每行中删除重复的单词，然后运行单词统计。
pig 步:

REGISTER 'udf-1.0-SNAPSHOT.jar'
define tuple_set com.ts.pig.UniqueRecords();
data = load '<file>' using PigStorage();

remove duplicate words from each line

unique= foreach data generate tuple_set($0) as line;
words= foreach unique generate flatten(TOKENIZE(line,' ')) as word;
grouped = group words BY word;
count= foreach grouped GENERATE group, COUNT(words);
dump count;

Pig UDF示例代码:

/**
 * This udf removes duplicate words from line
 */
public class UniqueRecords extends EvalFunc<String> {
    @Override
    public String exec(Tuple tuple) throws IOException {
        if (tuple == null || tuple.size() == 0)
            return null;
        String[] splits=tuple.get(0).toString().split(" ");
        Set<String> elements = new HashSet<String>(Arrays.asList(splits));
        StringBuilder sb = new StringBuilder();
        for(String element:elements ){
            sb.append(element+" ");
        }
        return sb.toString();
    }
}