apache-kafka - kafka-preferred-replica-election.sh 和 auto.leader.rebalance.enable 有什么区别？

Question

重启broker后，发现一个topic的leader的所有partition都被删除了 全部在经纪人 3 中，我已经设置 imbalance.check.interval.seconds 300，auto.leader.rebalance.enable True但 300 秒后什么也没发生。

所以我使用了 bin/kafka-preferred-replica-election.sh，得到了我期望的结果， 领导者被平衡为经纪人 1、2、3。

我想知道为什么没有发生自动重新平衡？ kafka-preferred-replica-election.sh 和 auto.leader.rebalance.enable 有什么区别？

Controller 日志:

[2019-08-14 09:31:33,454] INFO [Controller id=3] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [2019-08-14 09:31:33,454] TRACE [Controller id=3] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [2019-08-14 09:31:33,455] DEBUG [Controller id=3] Preferred replicas by broker Map(2 -> Map(__consumer_offsets-22 -> Vector(2), __consumer_offsets-4 -> Vector(2), __consumer_offsets-7 -> Vector(2), __consumer_offsets-46 -> Vector(2), __consumer_offsets-25 -> Vector(2), __consumer_offsets-49 -> Vector(2), __consumer_offsets-16 -> Vector(2), __consumer_offsets-28 -> Vector(2), __consumer_offsets-31 -> Vector(2), __consumer_offsets-37 -> Vector(2), __consumer_offsets-19 -> Vector(2), __consumer_offsets-13 -> Vector(2), fourth_topic-1 -> Vector(2, 3, 1), __consumer_offsets-43 -> Vector(2), __consumer_offsets-1 -> Vector(2), __consumer_offsets-34 -> Vector(2), __consumer_offsets-10 -> Vector(2), __consumer_offsets-40 -> Vector(2)), 1 -> Map(__consumer_offsets-30 -> Vector(1), __consumer_offsets-21 -> Vector(1), __consumer_offsets-27 -> Vector(1), __consumer_offsets-9 -> Vector(1), __consumer_offsets-33 -> Vector(1), __consumer_offsets-36 -> Vector(1), __consumer_offsets-42 -> Vector(1), __consumer_offsets-3 -> Vector(1), __consumer_offsets-18 -> Vector(1), __consumer_offsets-15 -> Vector(1), __consumer_offsets-24 -> Vector(1), __consumer_offsets-48 -> Vector(1), __consumer_offsets-6 -> Vector(1), fourth_topic-0 -> Vector(1, 2, 3), __consumer_offsets-0 -> Vector(1), __consumer_offsets-39 -> Vector(1), __consumer_offsets-12 -> Vector(1), __consumer_offsets-45 -> Vector(1)), 3 -> Map(__consumer_offsets-8 -> Vector(3), __consumer_offsets-35 -> Vector(3), __consumer_offsets-41 -> Vector(3), __consumer_offsets-23 -> Vector(3), __consumer_offsets-47 -> Vector(3), fourth_topic-2 -> Vector(3, 1, 2), __consumer_offsets-38 -> Vector(3), __consumer_offsets-17 -> Vector(3), __consumer_offsets-11 -> Vector(3), __consumer_offsets-2 -> Vector(3), __consumer_offsets-14 -> Vector(3), __consumer_offsets-20 -> Vector(3), __consumer_offsets-44 -> Vector(3), __consumer_offsets-5 -> Vector(3), __consumer_offsets-26 -> Vector(3), __consumer_offsets-29 -> Vector(3), __consumer_offsets-32 -> Vector(3))) (kafka.controller.KafkaController) [2019-08-14 09:31:33,455] DEBUG [Controller id=3] Topics not in preferred replica for broker 2 Map(fourth_topic-1 -> Vector(2, 3, 1)) (kafka.controller.KafkaController) [2019-08-14 09:31:33,455] DEBUG [Controller id=3] Topics not in preferred replica for broker 2 Map(fourth_topic-1 -> Vector(2, 3, 1)) (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] TRACE [Controller id=3] Leader imbalance ratio for broker 2 is 0.05555555555555555 (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] DEBUG [Controller id=3] Topics not in preferred replica for broker 1 Map(fourth_topic-0 -> Vector(1, 2, 3)) (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] TRACE [Controller id=3] Leader imbalance ratio for broker 1 is 0.05555555555555555 (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] DEBUG [Controller id=3] Topics not in preferred replica for broker 3 Map() (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] TRACE [Controller id=3] Leader imbalance ratio for broker 3 is 0.0 (kafka.controller.KafkaController)

但是 fourth_topic 分区的领导者仍然都在 broker 3 中:

bin/kafka-topics.sh --zookeeper 10.xx.xx.xx:2181/kafka  --topic fourth_topic --describe
Topic:fourth_topic      PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: fourth_topic     Partition: 0    Leader: 3       Replicas: 1,2,3 Isr: 3,1,2
        Topic: fourth_topic     Partition: 1    Leader: 3       Replicas: 2,3,1 Isr: 3,1,2
        Topic: fourth_topic     Partition: 2    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2

Answer 1

运行 kafka-preferred-replica-election.sh 时，它会强制为所有分区选择首选副本。

另一方面，当您将 auto.leader.rebalance.enable 设置为 true 时，Controller 将定期检查不平衡(每 leader.imbalance.check.interval.seconds)。但是，为了避免集群上不必要的负载，只有当不平衡率高于 leader.imbalance.per.broker.percentage(默认为 10%)时，leader 才会自动重新平衡。

您可以在 Controller 日志中看到当前的不平衡率(默认:${kafka.logs.dir}/controller.log)。