hadoop - 多个笛卡尔加入pySpark

Question

即使是很小的数据集，执行多个笛卡尔联接时也会出现内存错误。谁能解释为什么会这样？

In [1]: foo = sc.records([{'foo': 123}, {'foo': 321}])
In [2]: bar = sc.records([{'bar': 123}, {'bar': 321}])
In [3]: baz = sc.records([{'baz': 123}, {'baz': 321}])
In [4]: qux = foo.cartesian(bar)\
   ...:          .map(lambda (x,y): x.merge(y))\
   ...:          .cartesian(baz)\
   ...:          .map(lambda (x,y): x.merge(y))
In [5]: qux.collect()

java.lang.OutOfMemoryError: GC overhead limit exceeded

Answer 1

我最终定义了自己的cartesianJoin函数

def cartesianJoin(self, other):
    return self.map(lambda rec: (0, rec)).join(other.map(lambda rec: (0, rec))).map(lambda (key, (x, y)): x.merge(y))
end