scala - 我有一个以 Map 作为列数据类型的表，如何将其分解以生成 2 列，一列用于映射，一列用于键？

Question

Hive Table: (Name_Age: Map[String, Int] and ID: Int)

+---------------------------------------------------------++------+
|                  Name_Age                               ||  ID  | 
+---------------------------------------------------------++------+
|"SUBHAJIT SEN":28,"BINOY MONDAL":26,"SHANTANU DUTTA":35  ||  15  |
|"GOBINATHAN SP":35,"HARSH GUPTA":27,"RAHUL ANAND":26     ||  16  |
+---------------------------------------------------------++------+

我已将 Name_Age 列分解为多行:

def toUpper(name: Seq[String]) =  (name.map(a => a.toUpperCase)).toSeq

sqlContext.udf.register("toUpper",toUpper _)

var df = sqlContext.sql("SELECT toUpper(name) FROM namelist").toDF("Name_Age")

df.explode(df("Name_Age")){case org.apache.spark.sql.Row(arr: Seq[String]) => arr.toSeq.map(v => Tuple1(v))}.drop(df("Name_Age")).withColumnRenamed("_1","Name_Age")

+-------------------+
|     Name_Age      |
+-------------------+
|  [SUBHAJIT SEN,28]|
|  [BINOY MONDAL,26]|
|[SHANTANU DUTTA,35]|
| [GOBINATHAN SP,35]|
|   [HARSH GUPTA,27]|
|   [RAHUL ANAND,26]|
+-------------------+

但我想分解并创建 2 行:姓名和年龄

+-------------------+-------+
|       Name        |  Age  |
+-------------------+-------+
|  SUBHAJIT SEN     |  28   |
|  BINOY MONDAL     |  26   | 
|SHANTANU DUTTA     |  35   |
| GOBINATHAN SP     |  35   |
|   HARSH GUPTA     |  27   |
|   RAHUL ANAND     |  26   |
+-------------------+-------+

有人可以帮忙修改爆炸代码吗？

Answer 1

您所需要的只是删除 toUpper 调用 explode 函数:

import org.apache.spark.sql.functions.explode

val df = Seq((Map("foo" -> 1, "bar" -> 2), 1)).toDF("name_age", "id")
val exploded = df.select($"id", explode($"name_age")).toDF("id", "name", "age")
exploded.printSchema

// root
//  |-- id: integer (nullable = false)
//  |-- name: string (nullable = false)
//  |-- age: integer (nullable = false)

之后您可以使用内置函数转换为大写:

import org.apache.spark.sql.functions.upper

exploded.withColumn("name", upper($"name"))