python - 带有spyder的Anaconda : ImportError: cannot import name 'SparkConf'

Question

我在 anaconda 的 testenv 上安装了 pyspark (通过使用:conda install -c conda-forge pyspark)，它就在这里(我认为)

/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/pyspark/python/pyspark

这条路径存在，接下来我启动spyder:

(testenv1)➜~spyder

此代码产生以下错误，我认为 site-packeges 会自动“包含”，或者这是一个不同的问题？

import os
os.environ['SPARK_HOME'] = "/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/pyspark" # Not working but also not sure why I need to add this line at all pyspark appears to be in `site-packages`

from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local").setAppName("WordCount")
sc = SparkContext(conf = conf)

我收到以下错误:

runfile('/Users/myuser/dev/projects/python-snippets/pyspark.py', wdir='/Users/myuser/dev/projects/python-snippets')
Traceback (most recent call last):

  File "<ipython-input-1-969f4e596614>", line 1, in <module>
    runfile('/Users/myuser/dev/projects/python-snippets/pyspark.py', wdir='/Users/myuser/dev/projects/python-snippets')

  File "/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Users/myuser/dev/projects/python-snippets/pyspark.py", line 13, in <module>
    from pyspark import SparkConf, SparkContext

  File "/Users/myuser/dev/projects/python-snippets/pyspark.py", line 13, in <module>
    from pyspark import SparkConf, SparkContext

ImportError: cannot import name 'SparkConf'

请注意，我已尝试将spyder中的python解释器更新为:/Users/myuser/anaconda3/envs/testenv1/bin/python3.6，但我得到了同样的错误。

Answer 1

python-snippets/pyspark.py 是您的文件吗？如果是，则不应使用名称 pyspark.py，因为它会与原始 pyspark 包冲突。

请将文件重命名为其他名称，它应该可以工作