apache-kafka - 卡夫卡有重复的消息

Question

我在生成或使用数据时没有看到任何失败，但是在生产中有一堆重复的消息。对于一个获得大约 100k 条消息的小主题，有大约 4k 重复，尽管就像我说的没有失败，最重要的是没有实现重试逻辑或设置配置值。

我还检查了那些重复消息的偏移值，每个消息都有不同的值，告诉我问题出在生产者身上。

任何帮助将不胜感激

Answer 1

阅读有关 kafka 中消息传递的更多信息:

https://kafka.apache.org/08/design.html#semantics

So effectively Kafka guarantees at-least-once delivery by default and allows the user to implement at most once delivery by disabling retries on the producer and committing its offset prior to processing a batch of messages. Exactly-once delivery requires co-operation with the destination storage system but Kafka provides the offset which makes implementing this straight-forward.

可能您正在寻找像 jms 一样的“恰好一次交付”

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIgetexactly-oncemessagingfromKafka ?

There are two approaches to getting exactly once semantics during data production: 1. Use a single-writer per partition and every time you get a network error check the last message in that partition to see if your last write succeeded 2. Include a primary key (UUID or something) in the message and deduplicate on the consumer.

我们在我们的系统中实现了第二点。