apache-spark - 将 Spark-Streaming 的更改部署到生产的过程

Question

在不停机的情况下对 Spark-Streaming 中的生产进行一些更改的过程是什么？

Answer 1

如果您正在寻找升级应用程序代码，请参阅 spark-streaming 文档。

Upgrading Application Code If a running Spark Streaming application needs to be upgraded with new application code, then there are two possible mechanisms.

The upgraded Spark Streaming application is started and run in parallel to the existing application. Once the new one (receiving the same data as the old one) has been warmed up and is ready for prime time, the old one be can be brought down. Note that this can be done for data sources that support sending the data to two destinations (i.e., the earlier and upgraded applications).

The existing application is shutdown gracefully (see StreamingContext.stop(...) or JavaStreamingContext.stop(...) for graceful shutdown options) which ensure data that has been received is completely processed before shutdown. Then the upgraded application can be started, which will start processing from the same point where the earlier application left off. Note that this can be done only with input sources that support source-side buffering (like Kafka, and Flume) as data needs to be buffered while the previous application was down and the upgraded application is not yet up. And restarting from earlier checkpoint information of pre-upgrade code cannot be done. The checkpoint information essentially contains serialized Scala/Java/Python objects and trying to deserialize objects with new, modified classes may lead to errors. In this case, either start the upgraded app with a different checkpoint directory, or delete the previous checkpoint directory.

https://spark.apache.org/docs/latest/streaming-programming-guide.html