google-cloud-dataflow - Google Dataflow 延迟数据

Question

我一直在阅读 Dataflow SDK 文档，试图找出当数据到达流作业中的水印时会发生什么。

这一页:

https://cloud.google.com/dataflow/model/windowing

表示如果使用默认的 window/trigger 策略，则后期数据将被丢弃:

Note: Dataflow's default windowing and trigger strategies discard late data. If you want to ensure that your pipeline handles instances of late data, you'll need to explicitly set .withAllowedLateness when you set your PCollection's windowing strategy and set triggers for your PCollections accordingly.

然而这个页面:

https://cloud.google.com/dataflow/model/triggers

表示延迟数据在延迟到达时将作为单个元素 PCollection 发出:

The default trigger for a PCollection is event time-based, and emits the results of the window when the system's watermark (Dataflow's notion of when it "should" have all the data) passes the end of the window. The default trigger emits on a repeating basis, meaning that any late data will by definition arrive after the watermark and trip the trigger, causing the late elements to be emitted as they arrive.

那么，超过水印的后期数据会被完全丢弃吗？或者，它只是不会与其他数据一起发出，如果它及时到达，它会被窗口化，而是自己发出？

Answer 1

默认的“窗口和触发策略”丢弃迟到的数据。 WindowingStrategy是一个对象，它由窗口、触发和一些其他参数(例如允许的延迟)组成。默认允许的延迟为 0，因此任何延迟的数据元素都将被丢弃。

默认触发器处理延迟数据。如果你采用默认 WindowingStrategy并仅更改允许的迟到，然后您将收到 PCollection它包含所有准时数据的一个输出 Pane ，然后是大约每个延迟元素的新输出 Pane 。