gpt4 book ai didi

apache-kafka - Kafka Connect分布式模式组协调器不可用

转载 作者:行者123 更新时间:2023-12-04 03:19:34 25 4
gpt4 key购买 nike

我已经尝试了两个星期了,我在与连接节点不同的机器上运行 Kafka 集群。我无法正常运行连接。我可以读写kafka没问题。 Zookeeper 似乎运行良好。

我启动连接:

$ bin/connect-distributed connect-distributed.properties

Connect 不断循环遍历此错误:
[2018-08-21 15:45:12,161] INFO [Worker clientId=c1, groupId=connect-cluster] Discovered group coordinator 172.25.1.2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:677)
[2018-08-21 15:45:12,163] INFO [Worker clientId=c1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:509)
[2018-08-21 15:45:12,165] INFO [Worker clientId=c1, groupId=connect-cluster] Group coordinator 172.25.1.2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:729)
[2018-08-21 15:45:12,266] INFO [Worker clientId=c1, groupId=connect-cluster] Discovered group coordinator 172.25.40.219:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:677)
[2018-08-21 15:45:12,267] INFO [Worker clientId=c1, groupId=connect-cluster] Group coordinator 172.25.1.2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:729)

这是我的 connect-distributed.properties 的样子:
bootstrap.servers=172.25.1.2:9092,172.25.1.3:9092,172.25.1.4:9092
group.id=connect-cluster

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false

offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
offset.storage.partitions=3

config.storage.topic=connect-configs
config.storage.replication.factor=3

status.storage.topic=connect-status
status.storage.replication.factor=3
status.storage.partitions=3

offset.flush.interval.ms=10000

rest.host.name=172.25.1.5
rest.port=8083

heartbeat.interval.ms=3000
session.timeout.ms=30000
security.protocol=PLAINTEXT
client.id=c1

plugin.path=/usr/share/java

__Consumer_offsets 主题如下所示:
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic __consumer_offsets                                       
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 3,2
Topic: __consumer_offsets Partition: 1 Leader: 2 Replicas: 1,2,3 Isr: 3,2
Topic: __consumer_offsets Partition: 2 Leader: 2 Replicas: 1,2,3 Isr: 3,2
Topic: __consumer_offsets Partition: 3 Leader: 1 Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 4 Leader: 2 Replicas: 2 Isr: 2.... etc

最佳答案

在 Go 中编写连接器后,我遇到了同样的问题。我是被迫自己解决的。

当连接器连接到 kafka 时,它会自动写入主题和 __offset_topics .当连接器崩溃时,它会在这些表中作为协调器留下自己的痕迹。当新连接器启动时,它会在表中找到记录并尝试与协调器通信。协调器无法响应并且连接器永远不会工作。

您可以通过以下两种方式之一修复此问题:删除所有主题( connect-configsconnect-offsetsconnect-status__offset_topics )并重新启动集群。另一种方法是从主题中删除协调器,我目前不确定如何执行。

关于apache-kafka - Kafka Connect分布式模式组协调器不可用,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/51952398/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com