apache-spark - 如何在 spark Dataframe 中像 SQL 一样实现 EXISTS 条件

Question

我很想知道，如何以 spark Dataframe 的方式实现 sql like exists 子句。

Answer 1

LEFT SEMI JOIN 相当于 Spark 中的 EXISTS 函数。

val cityDF= Seq(("Delhi","India"),("Kolkata","India"),("Mumbai","India"),("Nairobi","Kenya"),("Colombo","Srilanka")).toDF("City","Country")

val CodeDF= Seq(("011","Delhi"),("022","Mumbai"),("033","Kolkata"),("044","Chennai")).toDF("Code","City")

val finalDF= cityDF.join(CodeDF, cityDF("City") === CodeDF("City"), "left_semi")