apache-spark - "TypeError: an integer is required (got type bytes)"在 Python 3.8 上导入 pyspark 时

Question

这个问题在这里已经有了答案:





How to fix 'TypeError: an integer is required (got type bytes)' error when trying to run pyspark after installing spark 2.4.4

(5 个回答)


4 个月前关闭。

创建 conda 环境:

conda create -y -n py38 python=3.8
conda activate py38

从 Pip 安装 Spark:

pip install pyspark
# Successfully installed py4j-0.10.7 pyspark-2.4.5

尝试导入pyspark:

python -c "import pyspark"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/dmitrii_deriabin/anaconda3/envs/py38/lib/python3.8/site-packages/pyspark/__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "/Users/dmitrii_deriabin/anaconda3/envs/py38/lib/python3.8/site-packages/pyspark/context.py", line 31, in <module>
    from pyspark import accumulators
  File "/Users/dmitrii_deriabin/anaconda3/envs/py38/lib/python3.8/site-packages/pyspark/accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "/Users/dmitrii_deriabin/anaconda3/envs/py38/lib/python3.8/site-packages/pyspark/serializers.py", line 72, in <module>
    from pyspark import cloudpickle
  File "/Users/dmitrii_deriabin/anaconda3/envs/py38/lib/python3.8/site-packages/pyspark/cloudpickle.py", line 145, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "/Users/dmitrii_deriabin/anaconda3/envs/py38/lib/python3.8/site-packages/pyspark/cloudpickle.py", line 126, in _make_cell_set_template_code
    return types.CodeType(
TypeError: an integer is required (got type bytes)

似乎 Pyspark 附带了 cloudpickle 的预打包版本在 Python 3.8 上存在一些问题的软件包，现在在 pip 版本上已解决(至少从 1.3.0 版开始)，但是 Pyspark 版本仍然损坏。有没有人遇到同样的问题/有没有运气解决这个问题？

Answer 1

你必须将你的 python 版本从 3.8 降级到 3.7，因为 pyspark 不支持这个版本的 python。