delta-lake - 如何更新 Delta 中分区列的值？

Question

我想知道，是否可以更新增量表分区列的“值”？

该表按特定列分区，现在我想更新该特定列的值。我可以这样做吗？

---

(在 slack 上找到)

Answer 1

使用 replaceWhere 选项。

引用官方文档 Replace table schema :

By default, overwriting the data in a table does not overwrite the schema. When overwriting a table using mode("overwrite") without replaceWhere, you may still want to overwrite the schema of the data being written. You replace the schema and partitioning of the table by setting the overwriteSchema option to true.

引用文章Selectively updating Delta partitions with replaceWhere :

Delta makes it easy to update certain disk partitions with the replaceWhere option.

replaceWhere is particularly useful when you have to run a computationally expensive algorithm, but only on certain partitions.