Azure Eventhub 消费者

Question

为什么我们需要 Azure 存储帐户上的 blob 容器用于 Eventhub 消费者客户端(我使用的是 python)。为什么我们不能像在 Kafka 中那样直接使用来自 Eventhub(Kafka 术语中的主题)的消息，或者可以通过其他方式来完成？

我正在关注下面链接的官方 Azure 文档: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send

Answer 1

您正在直接从事件中心使用消息。存储帐户不以任何方式用作中间步骤或类似的东西。相反，存储帐户用于 checkpointing :

Checkpointing is a process by which readers mark or commit their position within a partition event sequence. Checkpointing is the responsibility of the consumer and occurs on a per-partition basis within a consumer group. This responsibility means that for each consumer group, each partition reader must keep track of its current position in the event stream, and can inform the service when it considers the data stream complete.

If a reader disconnects from a partition, when it reconnects it begins reading at the checkpoint that was previously submitted by the last reader of that partition in that consumer group. When the reader connects, it passes the offset to the event hub to specify the location at which to start reading. In this way, you can use checkpointing to both mark events as "complete" by downstream applications, and to provide resiliency if a failover between readers running on different machines occurs. It's possible to return to older data by specifying a lower offset from this checkpointing process. Through this mechanism, checkpointing enables both failover resiliency and event stream replay.

总结一下:存储帐户用于存储有关读取器及其在分区内位置的信息。

您可以编写自己的自定义检查点存储实现，请参阅这个问题:Is there a way to store the azure Eventhub checkpoint to a remote bucket such as Google cloud bucket?