Scala - 使用 "endsWith"过滤数据帧

Question

给定一个 DataFrame :

 val df = sc.parallelize(List(("Mike","1986","1976"), ("Andre","1980","1966"), ("Pedro","1989","2000")))
      .toDF("info", "year1", "year2")
df.show

 +-----+-----+-----+
 | info|year1|year2|
 +-----+-----+-----+
 | Mike| 1986| 1976|
 |Andre| 1980| 1966|
 |Pedro| 1989| 2000|
 +-----+-----+-----+

我尝试过滤所有以 df 结尾的 6 值，但得到异常。
我试过了 :

  val filtered = df.filter(df.col("*").endsWith("6"))
  org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: ResolvedStar(info#20, year1#21, year2#22)

我也试过这个:

val filtered = df.select(df.col("*")).filter(_ endsWith("6"))
error: missing parameter type for expanded function ((x$1) => x$1.endsWith("6"))

如何解决？谢谢

Answer 1

我不太确定您要做什么，但据我了解:

val df = sc.parallelize(List(("Mike","1986","1976"), ("Andre","1980","1966"), ("Pedro","1989","2000"))).toDF("info", "year1", "year2")
df.show 
# +-----+-----+-----+
# | info|year1|year2|
# +-----+-----+-----+
# | Mike| 1986| 1976|
# |Andre| 1980| 1966|
# |Pedro| 1989| 2000|
# +-----+-----+-----+

val conditions = df.columns.map(df(_).endsWith("6")).reduce(_ or _)
df.withColumn("condition", conditions).filter($"condition" === true).drop("condition").show
# +-----+-----+-----+
# | info|year1|year2|
# +-----+-----+-----+
# |Andre| 1980| 1966|
# | Mike| 1986| 1976|
# +-----+-----+-----+