apache-kafka - Kafka 快速入门，adverted.host.name 给出 kafka.common.LeaderNotAvailableException

Question

我能够在一台 linux 机器上本地运行一个简单的单节点 Kafka (kafka_2.11-0.8.2.1)，但是当我尝试远程运行生产者时，我遇到了一些令人困惑的错误。

我正在遵循快速入门指南 http://kafka.apache.org/documentation.html#quickstart .我停止了 kafka 进程并删除了/tmp 中的所有 zookeeper 和 karma 文件。我在本地 10.0.0.0/24 网络上使用外部 IP 地址进行 NAT，所以我修改了 server.properties告诉zookeeper如何广播我的外部地址，按照https://medium.com/@thedude_rog/running-kafka-in-a-hybrid-cloud-environment-17a8f3cfc284 :

advertised.host.name=MY.EXTERNAL.IP

然后我运行这个:

$ bin/zookeeper-server-start.sh config/zookeeper.properties
--> ...
$ export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" # small test server!
$ bin/kafka-server-start.sh config/server.properties
--> ...

我在远程机器上为我的生产者打开了防火墙，并创建了一个新主题并进行了验证:

$ bin/kafka-topics.sh --create --zookeeper MY.EXTERNAL.IP:2181 --replication-factor 1 --partitions 1 --topic test123
--> Created topic "test123".
$ bin/kafka-topics.sh --list --zookeeper MY.EXTERNAL.IP:2181
--> test123

但是，我远程运行的生产者给了我错误:

$ bin/kafka-console-producer.sh --broker-list MY.EXTERNAL.IP:9092 --topic test123
--> [2015-06-16 14:41:19,757] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
My Test Message
--> [2015-06-16 14:42:43,347] WARN Error while fetching metadata [{TopicMetadata for topic test123 -> 

由于主题 [test123] 的 kafka.common.LeaderNotAvailableException}]，主题 test123 没有分区元数据:class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
-->(重复多次)

(我禁用了整个防火墙以确保这不是问题。)

karma-startup 中的 stdout 错误重复出现: [2015-06-16 20:42:42,768] INFO Closing socket connection to /MY.EXTERNAL.IP. (kafka.network.Processor)
controller.log 多次给我这个:

java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
    at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
[2015-06-16 20:44:08,128] INFO [Controller-0-to-broker-0-send-thread], Controller 0 connected to id:0,host:MY.EXTERNAL.IP,port:9092 for sending state change requests (kafka.controller.RequestSendThread)
[2015-06-16 20:44:08,428] WARN [Controller-0-to-broker-0-send-thread], Controller 0 epoch 1 fails to send request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:7;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:MY.EXTERNAL.IP,port:9092;PartitionState:(test123,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) to broker id:0,host:MY.EXTERNAL.IP,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread)

运行这个似乎表明在 0 处有一个领导者:

$ ./bin/kafka-topics.sh --zookeeper MY.EXTERNAL.IP:2181 --describe --topic test123
--> Topic:test123   PartitionCount:1    ReplicationFactor:1 Configs:
Topic: test123  Partition: 0    Leader: 0   Replicas: 0 Isr: 0

我重新运行了这个测试，我的 server.log 表明在 0 处有一个领导者:

...
[2015-06-16 21:58:04,498] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2015-06-16 21:58:04,642] INFO Registered broker 0 at path /brokers/ids/0 with address MY.EXTERNAL.IP:9092. (kafka.utils.ZkUtils$)
[2015-06-16 21:58:04,670] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-06-16 21:58:04,736] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

当我从生产者发送消息时，我在日志中看到此错误:

[2015-06-16 22:18:24,584] ERROR [KafkaApi-0] error when handling request Name: TopicMetadataRequest; Version: 0; CorrelationId: 7; ClientId: console-producer; Topics: test123 (kafka.server.KafkaApis)
kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
    at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)

我认为这意味着由于某种原因无法找到经纪人？我很困惑这是什么意思...

Answer 1

对于最新版本的 Kafka(在撰写本文时为 0.10.0)，您不想使用 advertised.host.name根本。事实上，即使 [documentation] 声明 advertised.host.name已经弃用。此外，Kafka 不仅会将它用作生产者/消费者的“广告”主机名，还会用作其他代理(在多代理环境中)......代理的不同(可能是内部)DNS ......而且您真的不想从事向个人添加条目的业务/etc/hosts经纪人(啊!)

因此，基本上，您希望代理使用内部名称，但仅对生产者和消费者使用外部 FQDN。为此，您将更新 advertised.listeners反而。