java - Spark Streaming 优雅关机

Question

我看到有一个选项可以在两个地方配置正常关机:

1. 在定义 spark conf 时:

"spark.streaming.stopGracefullyOnShutdown", "true".

1. 停止流上下文时:

JavaStreamingContext scc;

scc.stop(true,true)

这两个选项有什么区别？

谢谢

Answer 1

第一种方式

sparkConf.set(“spark.streaming.stopGracefullyOnShutdown","true") 

Setting this parameter to True in spark configuration ensures the proper graceful shutdown in new Spark version (1.4 onwards) applications. Also we should not use 1st explicit shutdown hook approach or call the ssc.stop method in the driver along with this parameter . We can just set this parameter, and then call methods ssc.start() and ssc.awaitTermination() . No need to call ssc.stop method. Otherwise application might hung during shutdown.

第二种方式scc.stop(true,true)

The scc.stop() method’s 1st boolean argument is for stopping the associated spark context while the 2nd boolean argument is for graceful shutdown of streaming context

这个已经明确提到here作者:chandan prakash

希望这对您有所帮助!