apache-spark - 创建 StreamingContext 时出错

Question

我打开 Spark shell

spark-shell --packages org.apache.spark:spark-streaming-kafka_2.10:1.6.0

然后我想创建一个流上下文

import org.apache.spark._
import org.apache.spark.streaming._


val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount").set("spark.driver.allowMultipleContexts", "true")
val ssc = new StreamingContext(conf, Seconds(1))

我遇到了异常:

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:

Answer 1

当您打开spark-shell时，已经创建了一个流上下文。它称为 sc，意味着您不需要创建配置对象。只需使用现有的 sc 对象即可。

val ssc = new StreamingContext(sc,Seconds(1))

我们将使用 val 代替 var