apache-kafka - 在 Kafka Async Producer 出现故障期间如何保证排序？

Question

如果我使用 Kafka Async 生产者，假设缓冲区中有 X 条消息。
当它们在客户端实际处理时，如果代理或特定分区关闭了一段时间，kafka 客户端将重试，如果消息失败，它是否会将特定消息标记为失败并继续下一条消息(这可能导致无序消息)？或者，它是否会使批处理中的剩余消息失败以保持顺序？

我接下来要维护排序，所以理想情况下希望 kafka 从失败的地方使批处理失败，所以我可以从失败点重试，我将如何实现？

Answer 1

就像它在关于重试的 kafka 文档中所说的那样

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries will potentially change the ordering of records because if two records are sent to a single partition, and the first fails and is retried but the second succeeds, then the second record may appear first.

因此，回答您的标题问题，没有 kafka 在异步发送下没有顺序保证。

我正在更新基于彼得戴维斯问题的答案。

我认为，如果您想以批处理模式发送，那么保护它的唯一方法是设置 max.in.flight.requests.per.connection=1但正如文档所说:

Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).