scala-ide - Spark 构建路径与不兼容的 Scala 版本 (2.10.0) 交叉编译

Question

当我尝试在 scala IDE 中执行 Sparksql 代码时，出现以下错误，有人可以帮我解决这个问题吗？

spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page. spark       Unknown Scala Version Problem
akka-remote_2.10-2.3.11.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
akka-slf4j_2.10-2.3.11.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.   spark       Unknown Scala Version Problem
breeze_2.10-0.11.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.   spark       Unknown Scala Version Problem
breeze-macros_2.10-0.11.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.    spark       Unknown Scala Version Problem
chill_2.10-0.5.0.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page. spark       Unknown Scala Version Problem
jackson-module-scala_2.10-2.4.4.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
json4s-ast_2.10-3.2.10.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.   spark       Unknown Scala Version Problem
json4s-core_2.10-3.2.10.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
json4s-jackson_2.10-3.2.10.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.   spark       Unknown Scala Version Problem
quasiquotes_2.10-2.0.1.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.   spark       Unknown Scala Version Problem
scalatest_2.10-2.2.5.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page. spark       Unknown Scala Version Problem
spark-catalyst_2.10-1.4.0.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.    spark       Unknown Scala Version Problem
spark-core_2.10-1.5.2-tests.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
spark-core_2.10-1.5.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.    spark       Unknown Scala Version Problem
spark-graphx_2.10-1.4.0.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
spark-launcher_2.10-1.5.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.    spark       Unknown Scala Version Problem
spark-mllib_2.10-1.4.0-tests.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page. spark       Unknown Scala Version Problem
spark-mllib_2.10-1.4.0.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.   spark       Unknown Scala Version Problem
spark-network-common_2.10-1.5.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
spark-network-shuffle_2.10-1.5.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page. spark       Unknown Scala Version Problem
spark-sql_2.10-1.4.0.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page. spark       Unknown Scala Version Problem
spark-streaming_2.10-1.5.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.   spark       Unknown Scala Version Problem
spark-unsafe_2.10-1.5.2.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
spire_2.10-0.7.4.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page. spark       Unknown Scala Version Problem
spire-macros_2.10-0.7.4.jar of spark build path is cross-compiled with an incompatible version of Scala (2.10.0). In case this report is mistaken, this check can be disabled in the compiler preference page.  spark       Unknown Scala Version Problem
The version of scala library found in the build path of spark (2.10.4) is prior to the one provided by scala IDE (2.11.7). Setting a Scala Installation Choice to match.    spark       Unknown Scala Version Problem

Answer 1

在您的项目中，您使用的是用不同 Scala 版本构建的 jar；确实从日志来看:

• Scala IDE 使用 Scala 2.11.7
• 使用 Scala 2.10 构建的 Apache Spark 1.5.2

您需要对齐 jar 版本。实际上，出于兼容性原因，Spark 1.5.2 预构建(可从 here 下载)已使用 Scala 2.10 构建(请参阅 here )。网站上有一条注释:

Note: Scala 2.11 users should download the Spark source package and build with Scala 2.11 support.

为了解决该问题，请在您的项目中使用与构建 Spark 相同的 Scala 版本。

我建议尝试在 Scala IDE 中切换到 Scala 2.10，它将解决该问题。

Eclipse + Maven

就我而言，我将 Eclipse 与 Scala IDE 和 Maven 结合使用，因此我以这种方式更新了 Maven 依赖项:

<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.10.6</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.5.2</version>
</dependency>

然后，我在 IDE 中更改了 Scala 版本:右键单击项目 -> Scala -> 设置 Scala 安装或右键单击 Scala 库容器 -> 属性 em>并选择Scala 2.10，如下所示