java - 为什么启动 StreamingContext 失败并显示 “IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute” ？

Question

我正在尝试使用 Twitter 作为源执行 Spark Streaming 示例，如下所示:

public static void main (String.. args) {

    SparkConf conf = new SparkConf().setAppName("Spark_Streaming_Twitter").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);       
        JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(2));      
        JavaSQLContext sqlCtx = new JavaSQLContext(sc);     


        String[] filters = new String[] {"soccer"};

        JavaReceiverInputDStream<Status> receiverStream = TwitterUtils.createStream(jssc,filters);



         jssc.start();
         jssc.awaitTermination();

}

但是我遇到了以下异常

Exception in thread "main" java.lang.AssertionError: assertion failed: No output streams registered, so nothing to execute
    at scala.Predef$.assert(Predef.scala:179)
    at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:158)
    at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:416)
    at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:437)
    at org.apache.spark.streaming.api.java.JavaStreamingContext.start(JavaStreamingContext.scala:501)
    at org.learning.spark.TwitterStreamSpark.main(TwitterStreamSpark.java:53)

有什么建议可以解决这个问题吗？

Answer 1

When an output operator is called, it triggers the computation of a stream.

如果 DStream 上没有输出运算符，则不会调用任何计算。基本上你需要在流上调用以下任何方法

print()
foreachRDD(func)
saveAsObjectFiles(prefix, [suffix])
saveAsTextFiles(prefix, [suffix])
saveAsHadoopFiles(prefix, [suffix])

http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations

如果需要，您也可以先应用任何转换，然后再输出函数。