python - 如何通过 bool 列过滤 Spark 数据帧？

Question

我创建了一个具有以下架构的数据框:

In [43]: yelp_df.printSchema()
root
 |-- business_id: string (nullable = true)
 |-- cool: integer (nullable = true)
 |-- date: string (nullable = true)
 |-- funny: integer (nullable = true)
 |-- id: string (nullable = true)
 |-- stars: integer (nullable = true)
 |-- text: string (nullable = true)
 |-- type: string (nullable = true)
 |-- useful: integer (nullable = true)
 |-- user_id: string (nullable = true)
 |-- name: string (nullable = true)
 |-- full_address: string (nullable = true)
 |-- latitude: double (nullable = true)
 |-- longitude: double (nullable = true)
 |-- neighborhoods: string (nullable = true)
 |-- open: boolean (nullable = true)
 |-- review_count: integer (nullable = true)
 |-- state: string (nullable = true)

我只想选择“打开”列为“真”的记录。我在 PySpark 中运行的以下命令不返回任何内容:

yelp_df.filter(yelp_df["open"] == "true").collect()

Answer 1

您比较的数据类型不正确。 open被列为 bool 值，而不是字符串，所以做 yelp_df["open"] == "true"不正确 - "true"是一个字符串。

相反你想做

yelp_df.filter(yelp_df["open"] == True).collect()

这正确地比较了 open 的值反对 bool 原语 True , 而不是非 bool 字符串 "true" .