apache-spark - RDD 和 Dataset 的不同默认值持续存在

Question

我试图找到一个很好的答案，说明为什么 RDD 的默认持久化是 MEMORY_ONLY 而 Dataset 是 MEMORY_AND_DISK。但是我找不到它。

有谁知道为什么默认持久性级别不同？

Answer 1

仅仅是因为 MEMORY_ONLY 很少有用 - 在实践中有足够的内存来存储所有需要的数据并不常见，所以你经常不得不驱逐一些 block 或缓存数据部分。

相比于 DISK_AND_MEMORY 将数据逐出到磁盘，因此没有缓存 block 丢失。

选择 MEMORY_AND_DISK 作为默认缓存模式的确切原因由 SPARK-3824 解释。 (默认情况下，Spark SQL 应缓存在 MEMORY_AND_DISK 中):

Spark SQL currently uses MEMORY_ONLY as the default format. Due to the use of column buffers however, there is a huge cost to having to recompute blocks, much more so than Spark core. Especially since now we are more conservative about caching blocks and sometimes won't cache blocks we think might exceed memory, it seems good to keep persisted blocks on disk by default.