python-3.x - 示例 Pyspark 程序返回 [WinError 2] 系统找不到文件

Question

这是我要运行的代码。我已经为 spark、hadoop、java 和 python 设置了路径。使用 Java 8、Spark 2.2.1 和 hadoop 2.7.5。

import random
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName('MyFirstStandaloneApp')
sc = SparkContext(conf=conf)
NUM_SAMPLES = 20
def inside(p):
   x, y = random.random(), random.random()
   return x*x + y*y < 1

count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
         .filter(inside).count()
print("Pi is roughly %f" % (4.0 * count / NUM_SAMPLES))

我收到的错误在这里:

Traceback (most recent call last):
File "sample1.py", line 4, in <module>
 sc = SparkContext(conf=conf)
File "C:\ProgramData\Anaconda3\lib\site-packages\pyspark\context.py", line 
    115, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "C:\ProgramData\Anaconda3\lib\site-packages\pyspark\context.py", line 
         283, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\ProgramData\Anaconda3\lib\site-packages\pyspark\java_gateway.py", 
         line 80, in launch_gateway
        proc = Popen(command, stdin=PIPE, env=env)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 709, in __init__
      restore_signals, start_new_session)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 997, 
     in_execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

Answer 1

解决方案我安装了两次 spark，一个来自 Apache 的独立版本，另一个来自 Anaconda，导致路径出现问题。