arrays - 将 Array[String] 的 Spark 列拆分为 String 列

Question

如果我有一个包含一列 Array[String] 的数据框:

scala> y.show
+---+----------+
|uid|event_comb|
+---+----------+
|  c|  [xx, zz]|
|  b|  [xx, xx]|
|  b|  [xx, yy]|
|  b|  [xx, zz]|
|  b|  [xx, yy]|
|  b|  [xx, zz]|
|  b|  [yy, zz]|
|  a|  [xx, yy]|
+---+----------+

如何拆分列 "event_comb"分成两列(例如 "event1" 和 "event2" )？

Answer 1

如果您的列类型是 list 或 Map 您可以使用 getItem 函数来获取值
getItem(Object key)

An expression that gets an item at position ordinal out of an array, or gets a value by key key in a MapType.

val data = Seq(
    ("c", List("xx", "zz")),
  ("b", List("xx", "xx")),
  ("b", List("xx", "yy")),
  ("b", List("xx", "zz")),
  ("b", List("xx", "yy")),
  ("b", List("xx", "zz")),
  ("b", List("yy", "zz")),
  ("a", List("xx", "yy"))
  ).toDF("uid", "event_comb")

  data.withColumn("event1", $"event_comb".getItem(0))
      .withColumn("event2", $"event_comb".getItem(1))
      .show(false)

输出:

+---+----------+------+------+
|uid|event_comb|event1|event2|
+---+----------+------+------+
|c  |[xx, zz]  |xx    |zz    |
|b  |[xx, xx]  |xx    |xx    |
|b  |[xx, yy]  |xx    |yy    |
|b  |[xx, zz]  |xx    |zz    |
|b  |[xx, yy]  |xx    |yy    |
|b  |[xx, zz]  |xx    |zz    |
|b  |[yy, zz]  |yy    |zz    |
|a  |[xx, yy]  |xx    |yy    |
+---+----------+------+------+