dataframe - 在 DataFrame 上描述 vs printSchema 方法

Question

我在 pyspark 中运行此代码，describe 和 printSchema 之间的输出差异令人困惑。请看下面的代码。

describe() 将分数列作为字符串给出，而当我不使用括号进行描述或使用 printSchema() 时，它会将分数列作为 int 给出——实际上是这样。

这是我的数据框。

>>> df.show()
+-------+------+-----+
|   name|course|score|
+-------+------+-----+
| fsdhfu|     a|   56|
| sdjjfd|     a|   57|
|kljsjlk|     b|   23|
|  udjkx|     b|   89|
|    ias|     c|   36|
| jksdkj|     c|   37|
|  usdkj|     d|   48|
+-------+------+-----+

使用描述:

>>> df2.describe()
DataFrame[summary: string, name: string, course: string, score: string]
>>> df2.describe
<bound method DataFrame.describe of DataFrame[name: string, course: string, score: int]>

使用 printSchema:

>>> df2.printSchema()
root
 |-- name: string (nullable = true)
 |-- course: string (nullable = true)
 |-- score: integer (nullable = true)

Answer 1

它们之间的区别在于 schema 为您提供有关列的信息，例如列的名称及其数据类型，而 describe 提供有关数据集的统计信息。以下是关于 describe 的 spark 文档:

/**
   * Computes basic statistics for numeric and string columns, including count, mean, stddev, min,
   * and max. If no columns are given, this function computes statistics for all numerical or
   * string columns.
   *
   * This function is meant for exploratory data analysis, as we make no guarantee about the
   * backward compatibility of the schema of the resulting Dataset. If you want to
   * programmatically compute summary statistics, use the `agg` function instead.
   *
   * {{{
   *   ds.describe("age", "height").show()
   *
   *   // output:
   *   // summary age   height
   *   // count   10.0  10.0
   *   // mean    53.3  178.05
   *   // stddev  11.6  15.7
   *   // min     18.0  163.0
   *   // max     92.0  192.0
   * }}}
   *
   * Use [[summary]] for expanded statistics and control over which statistics to compute.
   *