python - Spark MinMaxScaler 在数据帧上

Question

假设我有以下数据框

+---+-----+-------+
|day| time| result|                 
+---+-----+-------+
|  1| 6   |  0.5  |
|  1| 7   |  10.2 |
|  1| 8   |   5.7 |
|  2| 6   |  11.0 |
|  2| 10  |  22.3 |
+---+-----+-------+

我喜欢标准化每天的结果，同时保留属于每个结果的时间。我喜欢使用 MinMaxScaler 我假设我已将每天的值转换为密集向量，但我如何保留时间值？

Answer 1

I like to normalize the results (...) I like to use MinMaxScaler

这两个要求是相互排斥的。 MinMaxScaler 不能用于对组进行操作。您可以使用窗口函数

from pyspark.sql.functions import min, max, col
from pyspark.sql.window import Window

df = spark.createDataFrame(
    [(1, 6, 0.5), (1, 7, 10.2), (1, 8, 5.7), (2, 6, 11.0), (2, 10, 22.3)], 
    ("day", "time", "result"))

w = Window.partitionBy("day")

scaled_result = (col("result") - min("result").over(w)) / (max("result").over(w) - min("result").over(w))

df.withColumn("scaled_result", scaled_result).show()
# +---+----+------+------------------+                                            
# |day|time|result|     scaled_result|
# +---+----+------+------------------+
# |  1|   6|   0.5|               0.0|
# |  1|   7|  10.2|               1.0|
# |  1|   8|   5.7|0.5360824742268042|
# |  2|   6|  11.0|               0.0|
# |  2|  10|  22.3|               1.0|
# +---+----+------+------------------+

或分组、聚合和连接:

minmax_result = df.groupBy("day").agg(min("result").alias("min_result"), max("result").alias("max_result"))

minmax_result.join(df, ["day"]).select(
    "day", "time", "result", 
     ((col("result") - col("min_result")) / (col("max_result") - col("min_result"))).alias("scaled_result")
).show()
# +---+----+------+------------------+                                            
# |day|time|result|     scaled_result|
# +---+----+------+------------------+
# |  1|   6|   0.5|               0.0|
# |  1|   7|  10.2|               1.0|
# |  1|   8|   5.7|0.5360824742268042|
# |  2|   6|  11.0|               0.0|
# |  2|  10|  22.3|               1.0|
# +---+----+------+------------------+