azure - Spark 流访问 azure Blob

Question

我试图将我的 azure Blob存储注册到我的Spark Streaming中，但收到此代码和错误:-

码:-

SparkConf sparkConf = new SparkConf().setAppName("JavaNetworkWordCount");
        JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, Durations.seconds(1));
        ssc.textFileStream("wasb[s]://mycontainer@rtest.blob.core.windows.net/");
        ssc.start();
        ssc.awaitTermination();

不确定WASB链接的路径应该是什么

https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#address-files-in-azure-storage

链接说我应该给出一个路径，但是我的容器没有任何路径。图像直接存储在容器中。

错误:-

java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
    at scala.Predef$.require(Predef.scala:224)
    at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:163)
    at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:513)
    at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:573)
    at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:572)
    at org.apache.spark.streaming.api.java.JavaStreamingContext.start(JavaStreamingContext.scala:554)
    at org.bnr.process_panos.JavaNetworkWordCount.main(JavaNetworkWordCount.java:43)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

您可以使用相对路径或绝对路径。例如，可以使用以下一种方法来引用HDInsight群集随附的hadoop-mapreduce-examples.jar文件:

示例1 :wasb://mycontainer@myaccount.blob.core.windows.net/example/jars/hadoop-mapreduce-examples.jar

示例2: wasb:///example/jars/hadoop-mapreduce-examples.jar

Example3 :/example/jars/hadoop-mapreduce-examples.jar

如果没有在DStream上使用输出运算符，则会发生以下错误消息，而不会调用任何计算。您将需要在流上调用以下任何方法。

打印()

foreachRDD(功能)

saveAsObjectFiles(前缀，[后缀])

saveAsTextFiles(前缀，[后缀])

saveAsHadoopFiles(前缀，[后缀])

有关更多详细信息，请参见“http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations”。