scala - 两个分区数据帧之间的 Spark 共置连接

Question

对于以下两个 DataFrames 之间的连接在 Spark 1.6.0 中

val df0Rep = df0.repartition(32, col("a")).cache
val df1Rep = df1.repartition(32, col("a")).cache
val dfJoin = df0Rep.join(df1Rep, "a")
println(dfJoin.count)

这是否不仅共同分区而且共同定位？我知道对于 RDD，如果使用相同的分区器并在相同的操作中混洗，则连接将位于同一位置。但是数据帧呢？谢谢你。

Answer 1

[ https://medium.com/@achilleus/https-medium-com-joins-in-apache-spark-part-3-1d40c1e51e1c]

根据上面提供的文章链接Sort-Merge join是默认的join，想补充一点

For Ideal performance of Sort-Merge join, it is important that all rows having the same value for the join key are available in the same partition. This warrants for the infamous partition exchange(shuffle) between executors. Collocated partitions can avoid unnecessary data shuffle. Data needs to be evenly distributed n the join keys. The number of join keys is unique enough so that they can be equally distributed across the cluster to achieve the max parallelism from the available partitions