scala - Spark运行时错误-ClassDefNotFound:SparkConf

Question

在安装和构建Apache Spark(尽管有很多警告)之后，我们的Spark应用程序的编译(使用“sbt软件包”)已成功完成。但是，当尝试使用spark-submit脚本运行我们的应用程序时，会导致运行时错误，指出未找到SparkConf类定义。 SparkConf.scala文件存在于我们的系统上，但似乎未正确构建。关于如何解决这个问题的任何想法？

user@compname:~/Documents/TestApp$ /opt/Spark/spark-1.4.0/bin/spark-submit --master local[4] --jars /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar target/scala-2.11/sparkesingest_2.11-1.0.0.jar ~/Desktop/CSV/data.csv es-index localhost
Warning: Local jar /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar does not exist, skipping.
log4j:WARN No appenders could be found for logger (App).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/01 13:56:58 INFO SparkContext: Running Spark version 1.4.0
15/07/01 13:56:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/07/01 13:56:59 WARN Utils: Your hostname, compname resolves to a loopback address: 127.0.1.1; using [IP ADDRESS] instead (on interface eth0)
15/07/01 13:56:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/07/01 13:56:59 INFO SecurityManager: Changing view acls to: user
15/07/01 13:56:59 INFO SecurityManager: Changing modify acls to: user
15/07/01 13:56:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); users with modify permissions: Set(user)
15/07/01 13:56:59 INFO Slf4jLogger: Slf4jLogger started
15/07/01 13:56:59 INFO Remoting: Starting remoting
15/07/01 13:56:59 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@[IP ADDRESS]]
15/07/01 13:56:59 INFO Utils: Successfully started service 'sparkDriver' on port 34276.
15/07/01 13:56:59 INFO SparkEnv: Registering MapOutputTracker
15/07/01 13:56:59 INFO SparkEnv: Registering BlockManagerMaster
15/07/01 13:56:59 INFO DiskBlockManager: Created local directory at /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45
15/07/01 13:56:59 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/07/01 13:56:59 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/httpd-f4c3c67a-d058-4aba-bd65-5352feb5f12e
15/07/01 13:56:59 INFO HttpServer: Starting HTTP Server
15/07/01 13:56:59 INFO Utils: Successfully started service 'HTTP file server' on port 33599.
15/07/01 13:56:59 INFO SparkEnv: Registering OutputCommitCoordinator
15/07/01 13:56:59 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/07/01 13:56:59 INFO SparkUI: Started SparkUI at http://[IP ADDRESS]:4040
15/07/01 13:57:00 ERROR SparkContext: Jar not found at file:/opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar
15/07/01 13:57:00 INFO SparkContext: Added JAR file:/home/user/Documents/TestApp/target/scala-2.11/sparkesingest_2.11-1.0.0.jar at http://[IP ADDRESS]:33599/jars/sparkesingest_2.11-1.0.0.jar with timestamp 1435784220028
15/07/01 13:57:00 INFO Executor: Starting executor ID driver on host localhost
15/07/01 13:57:00 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44746.
15/07/01 13:57:00 INFO NettyBlockTransferService: Server created on 44746
15/07/01 13:57:00 INFO BlockManagerMaster: Trying to register BlockManager
15/07/01 13:57:00 INFO BlockManagerMasterEndpoint: Registering block manager localhost:44746 with 265.4 MB RAM, BlockManagerId(driver, localhost, 44746)
15/07/01 13:57:00 INFO BlockManagerMaster: Registered BlockManager
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(143840) called with curMem=0, maxMem=278302556
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 140.5 KB, free 265.3 MB)
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(12635) called with curMem=143840, maxMem=278302556
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 12.3 KB, free 265.3 MB)
15/07/01 13:57:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:44746 (size: 12.3 KB, free: 265.4 MB)
15/07/01 13:57:00 INFO SparkContext: Created broadcast 0 from textFile at Ingest.scala:159
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
    at org.elasticsearch.spark.rdd.CompatUtils.<clinit>(CompatUtils.java:20)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at org.elasticsearch.hadoop.util.ObjectUtils.loadClass(ObjectUtils.java:71)
    at org.elasticsearch.spark.package$.<init>(package.scala:14)
    at org.elasticsearch.spark.package$.<clinit>(package.scala)
    at build.Ingest$.main(Ingest.scala:176)
    at build.Ingest.main(Ingest.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 17 more
15/07/01 13:57:00 INFO SparkContext: Invoking stop() from shutdown hook
15/07/01 13:57:00 INFO SparkUI: Stopped Spark web UI at http://[IP ADDRESS]:4040
15/07/01 13:57:00 INFO DAGScheduler: Stopping DAGScheduler
15/07/01 13:57:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
15/07/01 13:57:00 INFO Utils: path = /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45, already present as root for deletion.
15/07/01 13:57:00 INFO MemoryStore: MemoryStore cleared
15/07/01 13:57:00 INFO BlockManager: BlockManager stopped
15/07/01 13:57:01 INFO BlockManagerMaster: BlockManagerMaster stopped
15/07/01 13:57:01 INFO SparkContext: Successfully stopped SparkContext
15/07/01 13:57:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
15/07/01 13:57:01 INFO Utils: Shutdown hook called
15/07/01 13:57:01 INFO Utils: Deleting directory /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd

这是build.sbt文件:

scalaVersion := "2.11.6"

name := "SparkEsIngest"

version := "1.0.0"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.4.0" % "provided",
    "org.apache.spark" %% "spark-streaming" % "1.4.0" % "provided",
    "org.apache.spark" %% "spark-sql" % "1.4.0" % "provided",
    "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10")
  )

if ( System.getenv("QUERY_ES_RESOURCE") != null) {
  println("[info] Using lib/es-hadoop-build-snapshot/ unmanagedBase dir")
  unmanagedBase <<= baseDirectory { base => base / "lib/es-hadoop-build-snapshot" }
} else {
  println("[info] Using lib/ unmanagedBase dir")
  unmanagedBase <<= baseDirectory { base => base / "lib" }
}

resolvers += "conjars.org" at "http://conjars.org/repo"

resolvers += "clojars" at "https://clojars.org/repo"

Answer 1

您提交的JAR中是否包含spark JAR？似乎您告诉sbt您正在提供jar，但如果您指望jar在机器中，那么我在sbt中看不到“uncompileJars in Compile + = file(...)”会建议您不要这样做，因为它会导致此类问题。

试试unrar您是JAR，看看是否有spark JAR，如果不使用sbt-assembly或您选择的其他工具。