scala - 如何在 Spark RDD 中获取 Avg 和 Sum

Question

假设我有 Spark 功能:

val group = whereRdd.map(collection => collection.getLong("location_id") -> collection.getInt("feel"))
  .groupByKey
  .map(grouped => grouped._1 -> grouped._2.toSet)

group.foreach(g => println(g))

我得到:

(639461796080961,Set(15))
(214680441881239,Set(5, 10, 25, -99, 99, 19, 100))
(203328349712668,Set(5, 10, 15, -99, 99, 15, 10))

是否可以向该函数添加一个Map()，并放置每个集合的avg和sum？例如:

(639461796080961,Map("data" -> Set(5, 10, 25, -99, 99, 19, 100), "avg" -> 22.71, "sum" -> 159))

Answer 1

我建议的一件事是使用 Tuple 或 case 类而不是 Map。我的意思大致是这样的:

case class Location(id: Long, values: Set[Int], sum: Int, avg: Double)

val group = whereRdd
  .map(collection => 
    collection.getLong("location_id") -> collection.getInt("feel"))
  .groupByKey
  .map{case (id, values) => {
    val set = values.toSet
    val sum = set.sum
    val mean = sum / set.size.toDouble
    Location(id, set, sum, mean)
  }}

相对于Map的最大优点是它保持类型的顺序。