gpt4 book ai didi

High velocity streaming data enrichment with low velocity/slow changing data(低速/慢变数据的高速流数据富集化)

翻译 作者:bug小助手 更新时间:2023-10-26 22:28:41 25 4
gpt4 key购买 nike



My system consists of

我的系统包括



  • High velocity telemetry data generated by IoT devices.

  • Relatively static/slow changing reference/lookup data - Alarm Rules


Each IoT device has 0 to 1 Alarm Rules. An Alarm Rule has average size of 1-2 KB.

每个物联网设备都有0到1条告警规则。告警规则的平均大小为1-2 KB。


Most Alarm Rules, once set, stay the same for weeks, months, or even a year or more.

大多数警报规则一旦设置,就会在几周、几个月、甚至一年或更长时间内保持不变。


Eventual consistency of Alarm Rules is also acceptable - if Alarm Rule is edited, it is acceptable for it to take effect in 15-30 minutes.

告警规则的最终一致性也是可以接受的-如果编辑了告警规则,15-30分钟后生效是可以接受的。


Question - What would be the best approach to enrich device telemetry stream with alarm rules?

问题-使用警报规则丰富设备遥测数据流的最佳方法是什么?


Option 1 - RichAsyncFunction + in memory cache

选项1-内存缓存中的RichAsyncFunction+


Each time I receive a telemetry message from device, I execute RichAsyncFunction. It first checks if in memory cache has Alarm Rule. If no Alarm Rule is not found in cache, a request is sent to database. Cache items expire in 30 minutes.

每次我从设备接收遥测消息时,我都会执行RichAsyncFunction。它首先检查内存缓存中是否有告警规则。如果缓存中没有找到告警规则,则向数据库发送请求。缓存项目将在30分钟后过期。


Option 2 - KeyedProcessFunction + state object

选项2-KeyedProcessFunction+状态对象


Same logic as with option 1. Except instead of using in memory cache, I store Alarm Rule for each IoT device into ValueState<> and periodically refresh it using ctx.timerService().register... scheduler (what happens if this gets called multiple times? will onTimer function also get triggered multiple times or just once?).

与选项1的逻辑相同。除了不在内存缓存中使用之外,我将每个物联网设备的警报规则存储到ValueState<>中,并使用ctx.timerService().注册...调度程序(如果多次调用会发生什么情况?OnTimer函数也会被触发多次还是只触发一次?)


Option 3 - CoProcessFunction/KeyedCoProcessFunction + 2 streams, one for telemetry, second for alarms

选项3-CoProcessFunction/KeyedCoProcessFunction+2个流,一个用于遥测,第二个用于报警


This option offers the highest throughput and lowest latency. I would consume Kafka Topic for alarm rules and update ValueState<> with the stream data.

此选项可提供最高的吞吐量和最低的延迟。告警规则使用Kafka主题,流数据更新ValueState<>。


What's stopping me from implementing this solution is Kafka Topic message retention time. By default Kafka messages have 7 day retention time.

阻止我实施这个解决方案的是Kafka主题消息保留时间。默认情况下,Kafka消息有7天的保留时间。


If I have alarm rule A for device B, and I send it to Kafka Topic, if the alarm rule A does not change over the next 7 days, the alarm rule will no longer be visible on 8th day. Basically, on 8th day, when consuming messages from device B, the system won't see any device alarm rules.

如果我有B设备的告警规则A,并将其发送到Kafka Theme,如果告警规则A在接下来的7天内没有变化,则在第8天将看不到该告警规则。基本上,在第8天,当消费来自B设备的消息时,系统不会看到任何设备告警规则。


I could increase retention time to longer period, but that does not seem like a reasonable time. Alternative would be external service that periodically emits all alarm rules to Kafka topic, say, every 6-7 days.

我可以将保留时间延长,但这似乎不是一个合理的时间。另一种选择是外部服务,定期向Kafka主题发出所有告警规则,比如每6-7天。


更多回答

You seem to be assuming the the ground truth for the rules must live outside of Flink. Why not keep the rule state in Flink (permanently)?

你似乎在假设规则的基本真相必须生活在Flink之外。为什么不将规则状态保持为Flink(永久)?

Wouldn't that still require the initial state and future updates to come from Kafka topic/sql database/external service? Or is it possible for my services (for example REST API) to "directly" talk to Flink and insert/update/delete state?

这是否仍然需要来自Kafka主题/SQL数据库/外部服务的初始状态和未来更新?或者,我的服务(例如rest API)是否可以“直接”与Flink对话并插入/更新/删除状态?

I would still use Kafka to decouple the external service from Flink, but the 7-day retention interval doesn't have to be a blocker, if you're willing to rely on checkpoints or savepoints to preserve the rules.

我仍然会使用Kafka来将外部服务与Flink分离,但如果你愿意依靠检查点或保存点来维护规则,7天的保留间隔不一定是一个拦截器。

优秀答案推荐
更多回答

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com