java - Kafka 如何存储每个主题的偏移量？

Question

在轮询 Kafka 时，我使用 subscribe() 函数订阅了多个主题。现在，我想设置我想要从每个主题读取的偏移量，而不是在每个主题的 seek() 和 poll() 之后重新订阅。在轮询数据之前对每个主题名称迭代调用seek() 是否会获得结果？Kafka 中如何准确存储偏移量？

每个主题我有一个分区，只有一个消费者可以读取所有主题。

Answer 1

How does Kafka store offsets for each topic?

Kafka 已经将 offset 存储从 zookeeper 转移到 kafka brokers。原因如下:

Zookeeper is not a good way to service a high-write load such as offset updates because zookeeper routes each write though every node and hence has no ability to partition or otherwise scale writes. We have always known this, but chose this implementation as a kind of "marriage of convenience" since we already depended on zk.

Kafka 将偏移量提交存储在主题中，当消费者提交偏移量时，kafka 将提交偏移量消息发布到“提交日志”主题，并保留将组/主题/分区映射到最新偏移量的内存结构用于快速检索。在此 page about offset management 中可以找到更多设计信息.

Now, I want to set the offset from which I want to read from each topic, without resubscribing after every seek() and poll() from a topic.

kafka 管理工具有一个新功能可以重置偏移量。

kafka-consumer-group.sh --bootstrap-server 127.0.0.1:9092 --group
      your-consumer-group **--reset-offsets** --to-offset 1 --all-topics --execute

有more options you can use .