apache-spark - 将展开内存传输到存储内存失败

Question

我得到这个 java.lang.AssertionError 与 Spark 。该错误并没有真正解释导致此错误的原因(至少对我而言)。
有关导致此错误的原因和解决此问题的步骤的任何帮助都会有所帮助。

Answer 1

Spark 必须先反序列化您的数据，然后才能被应用程序使用，这种反序列化的数据称为“展开内存”。在您的情况下，您的执行程序可能缺乏足够的 RAM 来容纳完全反序列化的数据。从源代码:

There are two reasons for store failed: First, the block is partially-unrolled; second, the block is entirely unrolled and the actual stored data size is larger than reserved, but we can't request extra memory

https://github.com/apache/spark/blob/9628aca68ba0821b8f3fa934ed4872cabb2a5d7d/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L260