gpt4 book ai didi

scala - Spark 提交时 ClassNotFoundException scala.runtime.LambdaDeserialize

转载 作者:行者123 更新时间:2023-12-04 13:56:59 27 4
gpt4 key购买 nike

我按照 https://spark.apache.org/docs/2.1.0/quick-start.html 上的 Scala 教程进行操作

我的 Scala 文件

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
def main(args: Array[String]) {
val logFile = "/data/README.md" // Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println(s"Lines with a: $numAs, Lines with b: $numBs")
sc.stop()
}
}

和 build.sbt
name := "Simple Project"

version := "1.0"

scalaVersion := "2.12.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.2.0"

我跑了 sbt package成功(已删除除 scala 源代码和 build.sbt 之外的所有内容,然后再次运行 sbt package)
[info] Loading project definition from /home/cpu11453local/workspace/testspark_scala/project
[info] Loading settings from build.sbt ...
[info] Set current project to Simple Project (in build file:/home/my_name/workspace/testspark_scala/)
[info] Packaging /home/my_name/workspace/testspark_scala/target/scala-2.12/simple-project_2.12-1.0.jar ...
[info] Done packaging.
[success] Total time: 1 s, completed Nov 8, 2017 12:15:24 PM

但是,当我运行 spark submit
$SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local[4] simple-project_2.12-1.0.jar 

我有错误
java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize

完整的 Spark 提交输出 gist

最佳答案

正如@Alexey 所说,将 Scala 版本更改为 2.11 解决了该问题。

生成.sbt

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.11"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"

请注意,Scala 版本必须与 Spark 匹配。
查看 artifactId,spark-core_2.11 表示它与 scala 2.11 兼容(不向后或向前兼容)

关于scala - Spark 提交时 ClassNotFoundException scala.runtime.LambdaDeserialize,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/47172122/

27 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com