apache-kafka - 使用 __consumer_offsets 杀死节点会导致消费者没有消息消费

Question

我有 3 个节点(nodes0、node1、node2)Kafka 集群(broker0、broker1、broker2)，复制因子为 2，Zookeeper(使用与 Kafka tar 打包的 Zookeeper)运行在不同的节点(节点 4)上。

我在启动zookeper之后启动了broker 0，然后启动了剩余的节点。从代理 0 日志中可以看出，它正在读取 __consumer_offsets，并且它们似乎存储在代理 0 上。以下是示例日志:

卡夫卡版本:kafka_2.10-0.10.2.0

    2017-06-30 10:50:47,381] INFO [GroupCoordinator 0]: Loading group metadata for console-consumer-85124 with generation 2 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 10:50:47,382] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from __consumer_offsets-41 in 23 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,382] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from __consumer_offsets-44 (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,387] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from __consumer_offsets-44 in 5 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,387] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from __consumer_offsets-47 (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,398] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from __consumer_offsets-47 in 11 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 10:50:47,398] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from __consumer_offsets-1 (kafka.coordinator.GroupMetadataManager)

此外，我可以在同一代理 0 日志中看到 GroupCoordinator 消息。

[2017-06-30 14:35:22,874] INFO [GroupCoordinator 0]: Preparing to restabilize group console-consumer-34472 with old generation 1 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:22,877] INFO [GroupCoordinator 0]: Group console-consumer-34472 with generation 2 is now empty (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:25,946] INFO [GroupCoordinator 0]: Preparing to restabilize group console-consumer-6612 with old generation 1 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:25,946] INFO [GroupCoordinator 0]: Group console-consumer-6612 with generation 2 is now empty (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:38,326] INFO [GroupCoordinator 0]: Preparing to restabilize group console-consumer-30165 with old generation 1 (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:35:38,326] INFO [GroupCoordinator 0]: Group console-consumer-30165 with generation 2 is now empty (kafka.coordinator.GroupCoordinator)
    [2017-06-30 14:43:15,656] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
    [2017-06-30 14:53:15,653] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)

在使用 kafka-console-consumer.sh 和 kafka-console- Producer.sh 测试集群的容错能力时，我发现在杀死代理 1 或Broker 2，消费者仍然可以接收来自生产者的新消息。重新平衡正确进行。

但是，终止代理 0 不会导致任意数量的消费者消费新消息或旧消息。下面是broker 0被kill前后topic的状态。

之前

Topic:test-topic    PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: test-topic   Partition: 0    Leader: 2   Replicas: 2,0   Isr: 0,2
    Topic: test-topic   Partition: 1    Leader: 0   Replicas: 0,1   Isr: 0,1
    Topic: test-topic   Partition: 2    Leader: 1   Replicas: 1,2   Isr: 1,2

之后

Topic:test-topic    PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: test-topic   Partition: 0    Leader: 2   Replicas: 2,0   Isr: 2
    Topic: test-topic   Partition: 1    Leader: 1   Replicas: 0,1   Isr: 1
    Topic: test-topic   Partition: 2    Leader: 1   Replicas: 1,2   Isr: 1,2

以下是代理 0 被终止后在消费者日志中看到的 WARN 消息

[2017-06-30 14:19:17,155] WARN Auto-commit of offsets {test-topic-2=OffsetAndMetadata{offset=4, metadata=''}, test-topic-0=OffsetAndMetadata{offset=5, metadata=''}, test-topic-1=OffsetAndMetadata{offset=4, metadata=''}} failed for group console-consumer-34472: Offset commit failed with a retriable exception. You should retry committing offsets. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-06-30 14:19:10,542] WARN Auto-commit of offsets {test-topic-2=OffsetAndMetadata{offset=4, metadata=''}, test-topic-0=OffsetAndMetadata{offset=5, metadata=''}, test-topic-1=OffsetAndMetadata{offset=4, metadata=''}} failed for group console-consumer-30165: Offset commit failed with a retriable exception. You should retry committing offsets. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

经纪人属性。其余默认属性保持不变。

broker.id=0
delete.topic.enable=true

auto.create.topics.enable=false
listeners=PLAINTEXT://XXX:9092
advertised.listeners=PLAINTEXT://XXX:9092
log.dirs=/tmp/kafka-logs-test1
num.partitions=3
zookeeper.connect=XXX:2181

生产者属性。其余默认属性保持不变。

bootstrap.servers=XXX,XXX,XXX
compression.type=snappy

消费者属性。其余默认属性保持不变。

zookeeper.connect=XXX:2181
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group

据我了解，如果节点持有/代理 GroupCoordinator 和 __consumer_offsets 死亡，那么尽管为分区选举了新的领导者，消费者也无法恢复正常操作。

我在 post 中看到类似的内容。这篇文章建议重新启动失效的代理节点。但是，尽管有更多节点，消息消费也会有延迟，直到生产环境中重新启动 Broker 0 为止。

问题一:如何缓解上述情况？

问题2:有没有办法将GroupCoordinator __consumer_offsets更改为另一个节点？

如有任何建议/帮助，我们将不胜感激。

Answer 1

检查 __consumer_offsets 主题的复制因子。如果不是 3，那就是你的问题了。

运行以下命令kafka-topics --zookeeper localhost:2181 --describe --topic __consumer_offsets并查看输出的第一行是否显示“ReplicationFactor:1”或“ReplicationFactor: 3英寸。

在尝试首先设置一个节点，然后以复制因子 1 创建该主题时，这是一个常见问题。稍后，当您扩展到 3 个节点时，您会忘记更改此现有主题的主题级别设置，因此即使您正在生成和使用的主题是容错的，偏移量主题仍然仅停留在代理 0 上。