scala - org/spark_project/guava/cache/CacheLoader 的 NoClassDefFoundError

Question

我正在尝试使用 Maven 在 Scala IDE for Eclipse 中运行 Spark 程序。但是，我在初始化 SparkConf 的行上收到 java.lang.NoClassDefFoundError 错误。我还尝试添加 Guava 14.0.1 的依赖项，但也没有解决我的问题。 :

Exception in thread "main" java.lang.NoClassDefFoundError: org/spark_project/guava/cache/CacheLoader
    at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:73)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:68)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:55)
    at oursparkapp2.SimpleApp$.main(SimpleApp.scala:8)
    at oursparkapp2.SimpleApp.main(SimpleApp.scala)
Caused by: java.lang.ClassNotFoundException:   org.spark_project.guava.cache.CacheLoader
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 5 more 

我尝试运行的scala程序(SimpleApp.scala)如下:

package oursparkapp2

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

object SimpleApp {
  def main(args:Array[String]) {
    val conf = new SparkConf().setAppName("Hlelo")
  }
}

我的pom.xml文件如下:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>ez.spark</groupId>
  <artifactId>oursparkapp2</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <dependencies>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.8.0</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.0</version>
        <scope>provided</scope>
    </dependency>

  </dependencies>
</project>

此外，使用 Spark-shell 命令，Spark 在我的终端上运行良好。

Answer 1

我通过添加依赖项解决了这个问题:

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-network-common_2.11</artifactId>
        <version>2.1.0</version>
    </dependency>