apache-kafka - 我们应该使用 max.poll.records 还是 max.poll.interval.ms 来处理在 kafka 消费者中需要更长时间处理的记录？

Question

我试图了解在 kafka 消费者中处理需要更长时间处理的记录的更好选择是什么？我进行了一些测试来理解这一点，并观察到我们可以通过修改 max.poll.records 来控制这一点。或 max.poll.interval.ms .

现在我的问题是，什么是更好的选择？请建议。

Answer 1

max.poll.records简单地定义单个调用中返回的最大记录数 poll() .

现在 max.poll.interval.ms定义对 poll() 的调用之间的延迟.

max.poll.interval.ms: The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member. For consumers using a non-null group.instance.id which reach this timeout, partitions will not be immediately reassigned. Instead, the consumer will stop sending heartbeats and partitions will be reassigned after expiration of session.timeout.ms. This mirrors the behavior of a static consumer which has shutdown.

我相信您可以调整两者以获得预期的行为。例如，您可以计算消息的平均处理时间。如果平均处理时间是 1 秒，而您有 max.poll.records=100那么您应该为轮询间隔留出大约 100+ 秒的时间。