python - 在 Apache Zeppelin 中调用 pandas_profiling.ProfileReport.to_widgets() 时出现 NotImplementedError

Question

我正在尝试使用 pandas_profiling 包自动描述 Apaceh Zeppelin 内部的一些数据帧。

我正在运行的代码是:

%pyspark

import sys
print(sys.version_info)

import numpy as np
print("numpy: ", np.__version__)
import pandas as pd
print("pandas: ", pd.__version__)
import pandas_profiling as pp
print("pandas_profiling: ", pp.__version__)

from pandas_profiling import ProfileReport

df = spark.sql("SELECT * FROM database.table")

profile = ProfileReport(df, title="Report: table")

profile.to_widgets()

我的结果是:

sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)
numpy:  1.19.5
pandas:  1.1.5
pandas_profiling:  3.1.0


Fail to execute line 19: profile.to_widgets()
Traceback (most recent call last):
  File "/tmp/1662648724242-0/zeppelin_python.py", line 158, in <module>
    exec(code, _zcUserQueryNameSpace)
  File "<stdin>", line 19, in <module>
  File "/usr/local/lib/python3.6/site-packages/pandas_profiling/profile_report.py", line 414, in to_widgets
    display(self.widgets)
  File "/usr/local/lib/python3.6/site-packages/pandas_profiling/profile_report.py", line 197, in widgets
    self._widgets = self._render_widgets()
  File "/usr/local/lib/python3.6/site-packages/pandas_profiling/profile_report.py", line 315, in _render_widgets
    report = self.report
  File "/usr/local/lib/python3.6/site-packages/pandas_profiling/profile_report.py", line 179, in report
    self._report = get_report_structure(self.config, self.description_set)
  File "/usr/local/lib/python3.6/site-packages/pandas_profiling/profile_report.py", line 166, in description_set
    self._sample,
  File "/usr/local/lib/python3.6/site-packages/pandas_profiling/model/describe.py", line 56, in describe
    check_dataframe(df)
  File "/usr/local/lib/python3.6/site-packages/multimethod/__init__.py", line 209, in __call__
    return self[tuple(map(self.get_type, args))](*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pandas_profiling/model/dataframe.py", line 10, in check_dataframe
    raise NotImplementedError()
NotImplementedError

有什么办法可以解决这个问题？有希望从 Zeppelin 内部解决它吗？

Answer 1

NotImplementedError 是从 check_dataframe 引发的:https://github.com/ydataai/pandas-profiling/blob/v3.1.0/src/pandas_profiling/model/dataframe.py#L10 . check_dataframe 使用 multimethod用于启用对函数的多参数分派(dispatch)，目前仅支持 Pandas DataFrames:https://github.com/ydataai/pandas-profiling/blob/v3.1.0/src/pandas_profiling/model/pandas/dataframe_pandas.py#L11 .在代码片段中，您提供了一个 Spark 数据帧(spark.sql(...) 的结果)，其中似乎没有任何用于动态调度的注册函数。如果您使用 toPandas 方法将 Spark 数据帧转换为 Pandas 数据帧，它应该调用正确的 check_dataframe 函数:

%pyspark

import sys
print(sys.version_info)

import numpy as np
print("numpy: ", np.__version__)
import pandas as pd
print("pandas: ", pd.__version__)
import pandas_profiling as pp
print("pandas_profiling: ", pp.__version__)

from pandas_profiling import ProfileReport

df = spark.sql("SELECT * FROM database.table").toPandas() 

profile = ProfileReport(df, title="Report: table")

profile.to_widgets()

或者，您可以尝试注册自己的函数来检查 Spark 数据帧，即；

from pandas_profiling.model.dataframe import check_dataframe
from pyspark.sql import DataFrame as SparkDataFrame
@check_dataframe.register
def spark_check_dataframe(df: SparkDataFrame):
   # do something here or just make it a `pass`

但报告逻辑中的下游函数可能不(并且很可能不)与 Spark 数据帧兼容。

如果由于数据规模或对 API 的舒适程度而希望继续使用 Spark 数据帧，还有另一种选择，即 spark-df-profiling它基于 pandas 分析，但专为处理 Spark 数据帧而构建。