apache-kafka - Kafka 主题的理想分区数

Question

我目前正在处理一个有 6 个 kafka-brokers 的设置，数据正从两个 producer 推送到我的 topic每秒大约 4000 条消息的速率，我有 5 个 Consumer 作为一个小组来处理这个主题。我的 kafka 主题 的理想分区数应该是多少？

如果 brokers/consumers/producer 也需要任何更改，请随时告诉我。

Answer 1

一般而言，分区越多，吞吐量就越大。但是，还有其他考虑因素，如您运行的硬件限制、是否使用压缩等。Confluent here 提供了足够好的信息。这让您深入了解可用于得出分区数量的粗略计算。

A rough formula for picking the number of partitions is based onthroughput. You measure the throughout that you can achieve on asingle partition for production (call it p) and consumption (call itc). Let’s say your target throughput is t. Then you need to have atleast max(t/p, t/c) partitions. The per-partition throughput that onecan achieve on the producer depends on configurations such as thebatching size, compression codec, type of acknowledgement, replicationfactor, etc.

此外对于消费者

The consumer throughput is often application dependent since itcorresponds to how fast the consumer logic can process each message

所以最好的方法是针对您自己的用例进行衡量和基准测试