scala - 如何更新现有 SparkSession 实例或在 spark-shell 中创建一个新实例？

Question

当我启动时 spark-shell ，它创建了一个 SparkSession 的实例.但是，我应该按如下方式创建它:

val spark = SparkSession.builder()
                        .config("es.nodes",elasticHost)
                        .config("es.port",elasticPort)
                        .config("es.nodes.wan.only","true")
                        .appName("Test")
                        .getOrCreate()

如何更新现有 spark在 spark-shell还是像我上面展示的那样创建新的？

Answer 1

您可以使用 SparkSession.conf.set 设置配置属性或创建另一个 SparkSession使用 SparkSession.newSession 的实例然后设置属性。

set(key: String, value: String): Unit Sets the given Spark runtime configuration property.

newSession(): SparkSession Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.

两种方式的工作方式(几乎)相同，区别在于您可以临时将属性设置为新值并同时使用 SparkSession s 同时。

// hello property is not set
scala> spark.conf.getOption("hello")
res1: Option[String] = None

scala> spark.conf.set("hello", "world")

// hello property is set
scala> spark.conf.getOption("hello")
res3: Option[String] = Some(world)

// create a new SparkSession (so you'll have two at the same time)
val ns = spark.newSession

// hello is not set in a new session
scala> ns.conf.getOption("hello")
res4: Option[String] = None

ns.conf.set("hello", "hello in another session")

scala> ns.conf.getOption("hello")
res8: Option[String] = Some(hello in another session)

// the value of hello in the initial SparkSession remains unchanged
scala> spark.conf.getOption("hello")
res9: Option[String] = Some(world)