apache-spark - 将 Spark Dataframe 转换为 Scala Map 集合

Question

我正在尝试找到将整个 Spark 数据帧转换为 scala Map 集合的最佳解决方案。最好的说明如下:

从这里开始(在 Spark 示例中):

val df = sqlContext.read.json("examples/src/main/resources/people.json")

df.show
+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

对于如下所示的 Scala 集合(Map of Maps):

val people = Map(
Map("age" -> null, "name" -> "Michael"),
Map("age" -> 30, "name" -> "Andy"),
Map("age" -> 19, "name" -> "Justin")
)

Answer 1

我认为你的问题没有意义——你最外面的Map ，我只看到你试图将值填充到其中 - 你需要在最外面有键/值对 Map 。话虽这么说:

val peopleArray = df.collect.map(r => Map(df.columns.zip(r.toSeq):_*))

会给你:

Array(
  Map("age" -> null, "name" -> "Michael"),
  Map("age" -> 30, "name" -> "Andy"),
  Map("age" -> 19, "name" -> "Justin")
)

那时你可以这样做:

val people = Map(peopleArray.map(p => (p.getOrElse("name", null), p)):_*)

这会给你:

Map(
  ("Michael" -> Map("age" -> null, "name" -> "Michael")),
  ("Andy" -> Map("age" -> 30, "name" -> "Andy")),
  ("Justin" -> Map("age" -> 19, "name" -> "Justin"))
)

我猜这确实是您想要的。如果你想将它们键入任意 Long索引，你可以这样做:

val indexedPeople = Map(peopleArray.zipWithIndex.map(r => (r._2, r._1)):_*)

这给你:

Map(
  (0 -> Map("age" -> null, "name" -> "Michael")),
  (1 -> Map("age" -> 30, "name" -> "Andy")),
  (2 -> Map("age" -> 19, "name" -> "Justin"))
)