python - 创建 Spark Session 抛出异常回溯

Question

我是 jupyter notebook 的新手，我正在尝试运行一个 pyspark 代码，其缩写如下:

import pyspark as ps
from pyspark.sql import SQLContext
from pyspark.sql import Row

spark = ps.sql.SparkSession.builder \
            .master("local") \
            .appName("Book Recommendation System") \
            .getOrCreate()

我在使用以下语句创建 pyspark session 时遇到错误:

• “这个 SparkContext 可能是一个现有的”
• “不要为现有的 SparkContext 更新 SparkConf，因为它由所有 session 共享”

完整的错误解释如下所示:

> ---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-21-cd9ecd052473> in <module>
----> 1 spark = ps.sql.SparkSession.builder.master("local").appName("Book").getOrCreate()
      2 
      3 sc = spark.sparkContext
      4 sqlContext = SQLContext(sc)

c:\program files (x86)\python\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
    184                             sparkConf.set(key, value)
    185                         # This SparkContext may be an existing one.
--> 186                         sc = SparkContext.getOrCreate(sparkConf)
    187                     # Do not update `SparkConf` for existing `SparkContext`, as it's shared
    188                     # by all sessions.

c:\program files (x86)\python\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
    369         with SparkContext._lock:
    370             if SparkContext._active_spark_context is None:
--> 371                 SparkContext(conf=conf or SparkConf())
    372             return SparkContext._active_spark_context
    373 

c:\program files (x86)\python\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    126                 " is not allowed as it is a security risk.")
    127 
--> 128         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    129         try:
    130             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

c:\program files (x86)\python\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
    318         with SparkContext._lock:
    319             if not SparkContext._gateway:
--> 320                 SparkContext._gateway = gateway or launch_gateway(conf)
    321                 SparkContext._jvm = SparkContext._gateway.jvm
    322 

c:\program files (x86)\python\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf, popen_kwargs)
    103 
    104             if not os.path.isfile(conn_info_file):
--> 105                 raise Exception("Java gateway process exited before sending its port number")
    106 
    107             with open(conn_info_file, "rb") as info:

Exception: Java gateway process exited before sending its port number

有人知道我该怎么办吗？我不知道有什么问题!

Answer 1

您不需要ps.sql。。试试这个。

import pyspark as ps
from pyspark.sql import SparkSession
from pyspark.sql import Row

spark = SparkSession.builder \
            .master("local") \
            .appName("Book Recommendation System") \
            .getOrCreate()