java - 应该包含/导入什么来识别我的 Spark -java 代码中的 "$"操作 join 函数？

Question

在 java8 中使用 Spark-sql-2.4.1v。我正在尝试加入两个数据集，如下所示:

computed_df.as('s).join(accumulated_results_df.as('f),$"s.company_id" === $"f.company_id","inner")

这在 databrick 的笔记本中运行良好。

But when I try to implement the same in my spark java code in my Ide. It wont recognize the "$" function/operator even after including

import static org.apache.spark.sql.functions.*;

那么应该如何在我的 Spark Java 代码中使用它呢？

谢谢

Answer 1

答案是org.apache.spark.sql.Column。请参阅This .

public class Column

...

df("columnName")            // On a specific DataFrame.
col("columnName")           // A generic column no yet associcated with a DataFrame.
col("columnName.field")     // Extracting a struct field
col("`a.column.with.dots`") // Escape `.` in column names.
$"columnName"               // Scala short hand for a named column.
expr("a + 1")               // A column that is constructed from a parsed SQL Expression.
lit("abc")                  // A column that produces a literal (constant) value.