scala - sbt 在运行 Spark hello world 代码时出错？

Question

运行 spark hello world 程序时出现以下错误。

[info] Updating {file:/C:/Users/user1/IdeaProjects/sqlServer/}sqlserver...[info] Resolving org.apache.spark#spark-core_2.12;2.1.1 ...[warn]  module not found: org.apache.spark#spark-core_2.12;2.1.1[warn] ==== local: tried[warn]   C:\Users\user1\.ivy2\local\org.apache.spark\spark-core_2.12\2.1.1\ivys\ivy.xml[warn] ==== public: tried[warn]   https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/2.1.1/spark-core_2.12-2.1.1.pom[warn] ==== local-preloaded-ivy: tried[warn]   C:\Users\user1\.sbt\preloaded\org.apache.spark\spark-core_2.12\2.1.1\ivys\ivy.xml[warn] ==== local-preloaded: tried[warn]   file:/C:/Users/user1/.sbt/preloaded/org/apache/spark/spark-core_2.12/2.1.1/spark-core_2.12-2.1.1.pom[info] Resolving jline#jline;2.14.3 ...[warn]  ::::::::::::::::::::::::::::::::::::::::::::::[warn]  ::          UNRESOLVED DEPENDENCIES         ::[warn]  ::::::::::::::::::::::::::::::::::::::::::::::[warn]  :: org.apache.spark#spark-core_2.12;2.1.1: not found[warn]  ::::::::::::::::::::::::::::::::::::::::::::::[warn][warn]  Note: Unresolved dependencies path:[warn]          org.apache.spark:spark-core_2.12:2.1.1 (C:\Users\user1\IdeaProjects\sqlServer\build.sbt#L7-8)[warn]            +- mpa:mpa_2.12:1.0[trace] Stack trace suppressed: run last *:update for the full output.[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.12;2.1.1: not found[error] Total time: 1 s, completed May 9, 2017 11:05:44 AM

Here is the build.sbt,

name := "Mpa"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"

我的 Spark webcome 消息。

Welcome to      ____              __     / __/__  ___ _____/ /__    _\ \/ _ \/ _ `/ __/  '_/   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1      /_/Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)Type in expressions to have them evaluated.Type :help for more information.

Update:

I changed the built.sbt to

name := "Mpa"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core_2.11" % "2.1.0"

但还是得到了

[信息] 更新 {file:/C:/Users/user1/IdeaProjects/sqlServer/}sqlserver...
[信息] 解决 org.apache.spark#spark-core_2.11_2.11;2.1.0 ...
[警告] 未找到模块:org.apache.spark#spark-core_2.11_2.11;2.1.0
[警告] ==== 本地:尝试过
[警告] C:\Users\user1\.ivy2\local\org.apache.spark\spark-core_2.11_2.11\2.1.0\ivys\ivy.xml
[警告] ==== 公开:试过
[警告] https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11_2.11/2.1.0/spark-core_2.11_2.11-2.1.0.pom
[警告] ==== 本地预加载 Ivy :尝试过
[警告] C:\Users\user1\.sbt\preloaded\org.apache.spark\spark-core_2.11_2.11\2.1.0\ivys\ivy.xml
[警告] ==== 本地预加载:已尝试
[警告] 文件:/C:/Users/user1/.sbt/preloaded/org/apache/spark/spark-core_2.11_2.11/2.1.0/spark-core_2.11_2.11-2.1.0.pom
[信息] 正在解决 jline#jline;2.12.1 ...
[警告]::::::::::::::::::::::::::::::::::::::::::::::
[警告]:: Unresolved 依赖关系::
[警告]::::::::::::::::::::::::::::::::::::::::::::::
[警告]::org.apache.spark#spark-core_2.11_2.11;2.1.0: 未找到
[警告]::::::::::::::::::::::::::::::::::::::::::::::
[警告]
[警告] 注意:未解析的依赖路径:
[警告] org.apache.spark:spark-core_2.11_2.11:2.1.0 (C:\Users\user1\IdeaProjects\sqlServer\build.sbt#L7-8)
[警告] +- mpa:mpa_2.11:1.0
[trace] 堆栈跟踪被抑制:最后运行 *:update 以获得完整输出。
[错误] (*:update) sbt.ResolveException: Unresolved 依赖项: org.apache.spark#spark-core_2.11_2.11;2.1.0: 未找到
[错误] 总时间:1 s，完成时间 2017 年 5 月 9 日 1:01:01 PM

Answer 1

您在 built.sbt 文件中有错误，您必须更改 %%至% :

name := "Mpa"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" % "spark-core" % "2.1.1"

%%要求 Sbt 将当前的 scala 版本添加到工件

您可以 spark-core_2.11与 %解决问题。

// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"

希望这可以帮助!