apache-spark - Pyspark 在 groupby 中创建字典

Question

在 pyspark 中是否可以在 groupBy.agg() 内创建字典？ ?这是一个玩具示例:

import pyspark
from pyspark.sql import Row
import pyspark.sql.functions as F

sc = pyspark.SparkContext()
spark = pyspark.sql.SparkSession(sc)

toy_data = spark.createDataFrame([
    Row(id=1, key='a', value="123"),
    Row(id=1, key='b', value="234"),
    Row(id=1, key='c', value="345"),
    Row(id=2, key='a', value="12"),
    Row(id=2, key='x', value="23"),
    Row(id=2, key='y', value="123")])

toy_data.show()

+---+---+-----+
| id|key|value|
+---+---+-----+
|  1|  a|  123|
|  1|  b|  234|
|  1|  c|  345|
|  2|  a|   12|
|  2|  x|   23|
|  2|  y|  123|
+---+---+-----+

这是 预期输出 :

---+------------------------------------
id |  key_value
---+------------------------------------
1  | {"a": "123", "b": "234", "c": "345"}
2  | {"a": "12", "x": "23", "y": "123"}
---+------------------------------------

======================================

我试过这个，但不起作用。

toy_data.groupBy("id").agg(
    F.create_map(col("key"),col("value")).alias("key_value")
)

这会产生以下错误:

AnalysisException: u"expression '`key`' is neither present in the group by, nor is it an aggregate function....

Answer 1

agg组件必须包含实际的聚合函数。解决这个问题的一种方法是结合 collect_list

Aggregate function: returns a list of objects with duplicates.

struct :

Creates a new struct column.

和 map_from_entries

Collection function: Returns a map created from the given array of entries.

这就是你要做的:

toy_data.groupBy("id").agg(
    F.map_from_entries(
        F.collect_list(
            F.struct("key", "value"))).alias("key_value")
).show(truncate=False)

+---+------------------------------+
|id |key_value                     |
+---+------------------------------+
|1  |[a -> 123, b -> 234, c -> 345]|
|2  |[a -> 12, x -> 23, y -> 123]  |
+---+------------------------------+