apache-spark - 使用 sparkmeasure 时无法调用 Java 对象

Question

我正在尝试使用 Sparkmeausre 检查我的 Pyspark 代码的性能。我在 Windows 10 上使用 Pycharm Community Edition，并正确配置了 Pyspark。我执行了“pip install sparkmeasure”并成功安装了 sparkmeasure。现在，当我尝试运行这段代码时。

from pyspark import  SparkConf , SparkContext
from pyspark.context import  SparkContext
from pyspark.sql.session import  SparkSession
from sparkmeasure import StageMetrics


sc = SparkContext(master = "local" , appName = "sparkdemo")
spark = SparkSession(sc)
sm = StageMetrics(spark)

我遇到了错误。

File "C:/Users/nj123/PycharmProjects/pythonProject/sparkdemo.py", line 9, in <module>
sm = StageMetrics(spark)
File "C:\Users\nj123\PycharmProjects\pythonProject\venv\lib\site- 
packages\sparkmeasure\stagemetrics.py", line 15, in __init__
self.stagemetrics = self.sc._jvm.ch.cern.sparkmeasure.StageMetrics(self.sparksession._jsparkSession)
TypeError: 'JavaPackage' object is not callable

如何解决这个错误并正确配置 sparkmeasure 到 Pycharm？

Answer 1

感谢@user238607。以下是我为解决此问题而执行的步骤。

1. First download Sparkmeasure jar file from Maven Central.

2. Then move this jar file to the spark jar folder. Mine location was, C:\Spark\spark-3.0.1-bin-hadoop2.7\jars

3. Now, Go to pycharm again, and rerun the same code.

Link to download the jar file.