spark-structured-streaming - Azure 事件中心流 : Does Checkpointing override setStartingPosition?

Question

如果我们像这样在 EventHub conf 中指定起始位置:

EventHubsConf(ConnectionStringBuilder(eventHubConnectionString).build)
  .setStartingPosition(EventPosition.fromStartOfStream)
or
  .setStartingPosition(EventPosition.fromEndOfStream)

并在 StreamWriter 中指定检查点位置

streamingInputDF
  .writeStream
  .option("checkpointLocation", checkpointLocation)
  ...

重启后，setStartingPosition 是否变得无关紧要，因为检查点始终用作开始读取的点？

谢谢。

Answer 1

重新启动 streamimg 查询时，将使用检查点文件中存储的偏移量信息。

有趣的是，structured streaming eventhubs integration guide 中并未具体提及这一点, 然而，在 DStreams guide它是:

"The connector fully integrates with the Structured Streaming checkpointing mechanism. You can recover the progress and state of you query on failures by setting a checkpoint location in your query. This checkpoint location has to be a path in an HDFS compatible file system, and can be set as an option in the DataStreamWriter when starting a query."

确保遵循 checkpoint recovery 上的一般指导.