scala - SparkSession.Builder 失败，错误代码 "A master URL must be set in your configuration": "spark.master" is set to "local"

Question

我有:

val sparkBuilder: SparkSession.Builder = SparkSession
  .builder
  .appName("CreateModelDataPreparation")
  .config("spark.master", "local")
implicit val spark: SparkSession = sparkBuilder.getOrCreate()

但是，当我运行我的程序时，我仍然得到:

org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)

按照其他 posts 中的建议，在 Main 方法中设置 SparkSession 。这些似乎并没有解决问题。

这与建议的重复项不同，因为我都尝试过:

  def main(argv: Array[String]): Unit = {
    import DeweyConfigs.implicits.da3wConfig

    val commandlineArgs: DeweyReaderArgs = processCommandLineArgs(argv)

    val sparkBuilder: SparkSession.Builder = SparkSession
      .builder
      .appName("CreateModelDataPreparation")
      .master("local")
    implicit val spark: SparkSession = sparkBuilder.config("spark.master", "local").getOrCreate()
    import spark.implicits._
    ...

和

  def main(argv: Array[String]): Unit = {
    import DeweyConfigs.implicits.da3wConfig

    val commandlineArgs: DeweyReaderArgs = processCommandLineArgs(argv)

    val sparkBuilder: SparkSession.Builder = SparkSession
      .builder
      .appName("CreateModelDataPreparation")
      .config("master", "local")
    implicit val spark: SparkSession = sparkBuilder.config("spark.master", "local").getOrCreate()
    import spark.implicits._
    ...

Answer 1

尝试在构建器上添加 .master("local") 而不是您提供的配置参数。

我本以为他们做了同样的事情，但我很确定后者是有效的。