java - pig - 我得到 "Error: Java heap space"和数十万元组

Question

我有三组按类型分隔的数据，通常每个 uid 只有几百个元组。但是(可能是由于某些错误)很少有 uid 具有最多 200000-300000 行数据。

当单个数据包中有太多元组时，StuffProcessor 有时会抛出堆空间错误。我应该如何解决这个问题？我可以以某种方式检查单个 uid 是否有 100000 多个元组，然后将数据拆分为更小的批处理？

我对 Pig 完全陌生，几乎不知道自己在做什么。

-- Create union of the three stuffs 
stuff = UNION stuff1, stuff2, stuff3;

-- Group data by uid
stuffGrouped = group stuff by (long)$0;

-- Process data
processedStuff = foreach stuffGrouped generate StuffProcessor(stuff);

-- Flatten the UID groups into single table
flatProcessedStuff = foreach processedStuff generate FLATTEN($0);

-- Separate into different datasets by type, these are all schemaless
processedStuff1 = filter flatProcessedStuff by (int)$5 == 9;
processedStuff2 = filter flatProcessedStuff by (int)$5 == 17;
processedStuff3 = filter flatProcessedStuff by (int)$5 == 20;

-- Store everything into separate files into HDFS
store processedStuff1 into '$PROCESSING_DIR/stuff1.txt';
store processedStuff2 into '$PROCESSING_DIR/stuff2.txt';
store processedStuff3 into '$PROCESSING_DIR/stuff3.txt';

Cloudera 集群应分配 4GB 堆空间

这实际上可能与 cloudera 用户有关，因为我无法对某些用户(piggy 用户与 hdfs 用户)重现此问题。

Answer 1

如果您的 UDF 并不真正需要同时查看属于某个键的所有元组，您可能需要实现 Accumulator接口(interface)以便小批量处理它们。您还可以考虑实现Algebraic接口(interface)以加快处理速度。

内置COUNT就是一个完美的例子。