java - Spark Streaming StreamingContext 错误

Question

我是一名 Java 老手，正在尝试学习 Scala + Spark Streaming。我下载了基于 Eclipse 的 Scala IDE + Spark 核心 jar + Spark Streaming jar 两个 2.10 并尝试了示例 - 我收到错误:

val ssc = new StreamingContext(conf, Seconds(1));

Description Resource Path Location Type bad symbolic reference. A signature in StreamingContext.class refers to term conf in package org.apache.hadoop which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling StreamingContext.class. Lab.scala /AirStream/src line 10 Scala Problem

我在这里错过了什么吗？所有 SparkContext 都没有错误，但 StreamingContext 一直出现此错误。

Answer 1

我遇到了大致相同的问题。这通常是我为 scala/spark 练习编写的 scala 类:

package practice.spark

import org.apache.spark.SparkContext._
import org.apache.spark._
import org.apache.spark.sql._

object SparkService {
  def sparkInit(sparkInstanceConfig: Configuration): SparkService = {
    val sparkConf = new SparkConf().setAppName(sparkInstanceConfig.appName)
    val instanceSpark = new SparkService(sparkConf)
    return instanceSpark
  }
}

class SparkService(sparkConf: SparkConf) {
  val sc = new SparkContext(sparkConf)
  val sql = new org.apache.spark.sql.SQLContext(sc)
}

在我的 eclipse 项目属性>Java 构建路径>库中，我有 jre8 库、scala 2.11 库、spark-core_2.11 和 spark-sql_2.11。我遇到了错误

Description Resource Path Location Type missing or invalid dependency detected while loading class file 'SparkContext.class'. Could not access term hadoop in package org.apache, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.) A full rebuild may help if 'SparkContext.class' was compiled against an incompatible version of org.apache. BinAnalysisNew Unknown Scala Problem

然后我添加了 hadoop-core jar 添加到我的 Java 构建路径，它解决了这个问题。我使用了那个 jar 的最新版本。

这个问题也可以通过使用 gradle 或其他构建工具来解决，这些构建工具将获取项目中使用的每个 jar 的所有依赖项。