apache-spark - 无法在 HDP 2.5.3 上手动启动 Spark 2.1.0

Question

我在 Spark 2.0.0 上测试我的 Spark 代码，遇到了错误 SPARK-17463，我想使用 Spark 2.1.0，因为该错误已在此版本中修复。

但是，我无法在 Spark 2.1.0 中使用 yarn 客户端模式启动 Spark-Shell

我需要让 2.1.0 在 HDP 2.5.3 集群上运行。

它抛出异常:

    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/15 14:28:46 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
    java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
      at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
      at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
      at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
      at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
      at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
      ... 47 elided
    Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 61 more
<console>:14: error: not found: value spark
           import spark.implicits._
                          ^
<console>:14: error: not found: value spark
           import spark.sql
                          ^

Answer 1

我在 EMR 中遇到了与 Spark-submit 相同的问题，将 jersey-bundle-1.19.1.jar 添加到 $SPARK_HOME/jars 后，问题得到了解决。

您可以从这里下载:http://repo1.maven.org/maven2/com/sun/jersey/jersey-bundle/1.19.1/jersey-bundle-1.19.1.jar