apache-spark - 如何在 PySpark 中过滤 MapType 中的键？

Question

给定如下的 DataFrame 是否可以在保持架构完整的同时过滤掉 PySpark 中 Column 集合的某些键(MapType(StringType, StringType, True))？

root
 |-- id: string (nullable = true)
 |-- collection: map (nullable = true)
 |    |-- key: string
 |    |-- value: string

Answer 1

是的，这是可能的。您应该创建 udf负责从 map 中过滤键并与 withColumn 一起使用转换为来自 collection 的过滤键 field 。

下面是 Scala 中的示例实现:

// Start from implementing method in Scala responsible for filtering keys from Map
def filterKeys(collection: Map[String, String], keys: Iterable[String]): Map[String, String] =
    collection.filter{case (k,_) => !keys.exists(_ == k)}

// Create Spark UDF based on above function
val filterKeysUdf = udf((collection: Map[String, String], keys: Iterable[String]) => filterKeys(collection, keys))

// Use above udf to filter keys
val newDf = df.withColumn("collection", filterKeysUdf(df("collection"), lit(Array("k1"))))

在 Python 中的实现:

# Start from implementing method in Python responsible for filtering keys from dict
def filterKeys(collection, keys):
    return {k:collection[k] for k in collection if k not in keys}

# Create Spark UDF based on above function
filterKeysUdf = udf(filterKeys, MapType(StringType(), StringType()))

# Create array literal based on Python list
keywords_lit = array(*[lit(k) for k in ["k1","k2"]])

# Use above udf to filter keys
newDf = df.withColumn("collection", filterKeysUdf(df.collection, keywords_lit))