docker - Zookeeper重新启动后，Kafka群集丢失消息

Question

我正在使用Docker启动一个kafka代理集群(例如5个代理，每个容器一个代理)。 Kafka版本2.12-0.11.0.0，Zookeeper 3.4.10。

场景:

使用

以下的配置启动第一个代理

zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5

dataDir=/opt/zookeeper/data

clientPort=2181
maxClientCnxns=10
minSessionTimeout=4000
maxSessionTimeout=1000000
server.1=0.0.0.0:2888:3888

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker1_IP:broker1_PORT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=127.0.0.1:2181
zookeeper.session.timeout.ms=6000
zookeeper.connection.timeout.ms=1000000
group.initial.rebalance.delay.ms=0

producer.properties

bootstrap.servers=localhost:9092
compression.type=none

消费者属性

zookeeper.connect=127.0.0.1:2181
zookeeper.session.timeout.ms=6000
zookeeper.connection.timeout.ms=1000000
group.id=test-consumer-group

Zookeeper在独立模式下启动，然后启动kafka

创建主题

/opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-test-topic1

发送消息

echo "test_kafka1" | /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-test-topic1

正在检查消息

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-test-topic1 --max-messages 1
已收到消息

描述主题

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-test-topic1
Topic:my-test-topic1 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: my-test-topic1 Partition: 0 Leader: 1 Replicas: 1 Isr: 1

开始休息4个经纪人

每个经纪人在第1到第5位上的zoo.cfg(仅0.0.0.0:2888:3888位置不同)

tickTime=2000
initLimit=10
syncLimit=5

dataDir=/opt/zookeeper/data

clientPort=2181
maxClientCnxns=10
minSessionTimeout=4000
maxSessionTimeout=1000000
server.1=0.0.0.0:2888:3888
server.2=broker2_IP:broker2_2888:broker2_3888
server.3=broker3_IP:broker3_2888:broker3_3888
server.4=broker4_IP:broker4_2888:broker4_3888
server.5=broker5_IP:broker5_2888:broker5_3888

每个代理在第1到第5个服务器上的server.properties(broker.id是唯一的，ech_broker的broker_IP:broker_PORT不同)

broker.id=N
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker_IP:broker_PORT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=127.0.0.1:2181
zookeeper.session.timeout.ms=6000
zookeeper.connection.timeout.ms=1000000
group.initial.rebalance.delay.ms=0

每个经纪人从1日到5日的producer.properties

bootstrap.servers=localhost:9092
compression.type=none

每个经纪人从1日到5日的Consumer.properties

zookeeper.connect=127.0.0.1:2181
zookeeper.session.timeout.ms=6000
zookeeper.connection.timeout.ms=1000000
group.id=test-consumer-group

在每个代理上重新启动zookeeper，以使zoo.cfg生效

动物园管理员聚集成簇

主题已移至代理5

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-test-topic1
Topic:my-test-topic1 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: my-test-topic1 Partition: 0 Leader: 5 Replicas: 5 Isr: 5
这是正常行为吗？还是应该留在经纪人1上？

检查每个代理上的消息

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-test-topic1 --max-messages 1
消息丢失(当主题停留在代理1上时消息不会丢失，因此
这是 float 的情况)

Answer 1

您是否尝试将滴答时间提高到6000？根据Hadoop的设置，他们默认使用此设置，并指出2000毫秒设置太低。我认为这里也适用。我现在正在处理一个非常类似的kafka问题。