apache-kafka - 多个消费者从单个 kafka 分区消费

Question

我在 kafka docs 中阅读了以下内容:

• The way consumption is implemented in Kafka is by dividing up the partitions in the log over the consumer instances so that each instance is the exclusive consumer of a "fair share" of partitions at any point in time.
• Kafka only provides a total order over records within a partition, not between different partitions in a topic.
• Per-partition ordering combined with the ability to partition data by key is sufficient for most applications.
• However, if you require a total order over records this can be achieved with a topic that has only one partition, though this will mean only one consumer process per consumer group.

我阅读了以下内容 on this page :

• Consumers read from any single partition, allowing you to scale throughput of message consumption in a similar fashion to message production.
• Consumers can also be organized into consumer groups for a given topic — each consumer within the group reads from a unique partition and the group as a whole consumes all messages from the entire topic.
• If you have more consumers than partitions then some consumers will be idle because they have no partitions to read from.
• If you have more partitions than consumers then consumers will receive messages from multiple partitions.
• If you have equal numbers of consumers and partitions, each consumer reads messages in order from exactly one partition.

疑问

这是否意味着单个分区不能被多个消费者消费？我们不能有一个分区和一个有多个消费者的消费者组，并让他们都从单个分区消费？

如果单个分区只能被单个消费者消费，我在想为什么这个设计决定？

如果我需要对记录进行总订单并且仍然需要并行使用它怎么办？它在 Kafka 中是可撤销的吗？或者这样的场景没有意义？

Answer 1

在一个消费者组内，任何时候一个分区只能被一个消费者消费。不，您不能在同一组内有 2 个消费者同时从同一分区消费。

Kafka 消费者组允许多个消费者“有点”表现得像一个实体。整个组应该只消费一次消息。如果一个组中的多个消费者要消费相同的分区，这些记录将被多次处理。

如果您需要多次消费一个分区，请确保这些消费者在不同的组中。

当处理需要在任何时候按顺序(串行)发生时，只有一个任务要做。如果您有记录 1、2 和 3 并希望按顺序处理它们，则在处理消息 1 之前您无法执行任何操作。消息 2 和消息 3 是一样的。那么你想并行做什么？