pandas - AWS EMR : 'JavaPackage' object is not callable 上带有 Pandas 和 pyarrow 错误的 pyspark

Question

我正在尝试将 Pandas 数据帧转换为 Pyspark 数据帧，并收到以下与 pyarrow 相关的错误:

import pandas as pd
import numpy as np

data = np.random.rand(1000000, 10)
pdf = pd.DataFrame(data, columns=list("abcdefghij"))
df = spark.createDataFrame(pdf)

/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py:714: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true; however, failed by the reason below:
  'JavaPackage' object is not callable
Attempting non-optimization as 'spark.sql.execution.arrow.fallback.enabled' is set to true.

我尝试了不同版本的 pyarrow(0.10.0、0.14.1、0.15.1 和更多)，但结果相同。我该如何调试？

Answer 1

我遇到了同样的问题，将集群设置更改为 emr-5.30.1，将箭头版本更改为 0.14.1 并解决了问题