Scala Spark 版本兼容性

Question

我正在尝试在 IntelliJ IDE 中配置 Scala

我机器上的 Scala 和 Spark 版本

Welcome to Scala 2.12.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121).

apache-spark/2.2.1

SBT文件

scalaVersion := "2.12.5"
resolvers  += "MavenRepository" at "http://central.maven.org/maven2"

libraryDependencies ++= {
  val sparkVersion = "2.2.1"
    Seq( "org.apache.spark" %% "spark-core" % sparkVersion)
}

我收到错误

Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving jline#jline;2.14.5 ...
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.12;2.2.1: not found
[error] unresolved dependency: org.apache.spark#spark-core_2.12;1.4.0: not found
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.12;2.2.1: not found
[error] unresolved dependency: org.apache.spark#spark-core_2.12;1.4.0: not found

Answer 1

没有您在 sbt 项目中定义的 spark 核心版本可供下载。你可以查看maven dependency有关可用版本的更多信息

可以看到spark-core 2.2.1版本，要下载的最新版本是Scala 2.11编译的 info here

所以

或者您将 sbt 构建文件更改为

scalaVersion := "2.11.8"
resolvers  += "MavenRepository" at "http://central.maven.org/maven2"

libraryDependencies ++= {
  val sparkVersion = "2.2.1"
    Seq( "org.apache.spark" %% "spark-core" % sparkVersion)
}

或将构建依赖的版本定义为

libraryDependencies ++= {
  val sparkVersion = "2.2.1"
    Seq("org.apache.spark" % "spark-core_2.11" % sparkVersion)
}

希望回答对你有帮助