java - 使用 Spark 和 Java8 从 Dataframe 获取多列的不同值计数

Question

我想使用 Spark 和 Java8 从 Dataframe 中获取多列的不同值的计数

输入数据帧 - 需要为动态列编写代码 - 列可能稍后添加

+----+----+----+
|Col1|Col2|Col3|
+----+----+----+
|A1|Y|B2|Y|C3|Y|
|A1|Y|B2|N|C3|Y|
|A1|Y|B2|Y|C3|N|
+----+----+----+

输出日期帧

+--------+---------------------+--------------------+
|Col1    | Col2                | Col3               |
+--------+---------------------+--------------------+
|A1|Y - 3| B2|Y - 2 & B2|N - 1 | C3|Y - 3 & C3|N -1 |
+----+----+----+----+----+----+----+----+----+------+

Answer 1

也许这可以帮助你一点，在scala中使用rdd，但在java中应该非常相似。

  val df = Seq(("a", "a", "a"), ("a", "b", "c"), ("b", "b", "c")).toDF("Col1","Col2","Col3")
  df.show()

  val ok = df.rdd.map(s => {
    var arr = new Array[(String, String)](s.size)
    for (i <- 0 to s.size - 1) {
      arr(i) = (s.getString(i), s.schema.fieldNames(i))
    }
    arr
  }).map(s => {
    for (i <- s) yield ((i._2, i._1), 1)
  }).flatMap(s => s)
    .reduceByKey(_ + _)
    .map(s => (s._1._1, s._1._2 + "=" + s._2))
    .reduceByKey(_ +","+ _)

  ok.foreach(println(_))

 +----+----+----+
 |Col1|Col2|Col3|
 +----+----+----+
 |   a|   a|   a|
 |   a|   b|   c|
 |   b|   b|   c|
 +----+----+----+

 (Col1,a=2,b=1)
 (Col2,b=2,a=1)
 (Col3,a=1,c=2)