scala - Spark 结构化流多个 WriteStreams 到同一个接收器

Question

两个Writestream在 Spark Structured Streaming 2.2.1 中，同一个数据库接收器不会按顺序发生。请建议如何使它们按顺序执行。

val deleteSink = ds1.writestream
  .outputMode("update")
  .foreach(mydbsink)
  .start()

val UpsertSink = ds2.writestream
  .outputMode("update")
  .foreach(mydbsink)
  .start()

deleteSink.awaitTermination()
UpsertSink.awaitTermination()

使用上面的代码， deleteSink在 UpsertSink 之后执行.

Answer 1

如果你想让两个流并行运行，你必须使用

sparkSession.streams.awaitAnyTermination()

代替

deleteSink.awaitTermination()
UpsertSink.awaitTermination()

在您的情况下 UpsertSink 将永远不会启动，除非 deleteSink 将被停止或抛出异常，正如它在 scaladoc 中所说的那样

Waits for the termination of this query, either by query.stop() or by an exception. If the query has terminated with an exception, then the exception will be thrown. If the query has terminated, then all subsequent calls to this method will either return immediately (if the query was terminated by stop()), or throw the exception immediately (if the query has terminated with exception).