docker - 创建主题，但在Kubernetes上使用Python获得Kafka FailedPayloadsError

Question

我在python kafka库中使用SimpleProducer。
该脚本以前可以与我尝试过的其他更难配置的kafka设置完美配合。

kafka = KafkaClient(u'[masterNodeIp]:[servicePort]')
producer = SimpleProducer(kafka)
#make a simple message, while true run
producer.send_messages(b'oneMoreTopic', sentence)

一次运行此脚本后，我会在python-console中收到此响应。

kafka.common.LeaderNotAvailableError: TopicMetadata(topic='oneMoreTopic', error=5, partitions=[])

然后，我可以在zookeeper.log上进入我的Node并查看:

2015-09-14 12:16:32,276 - INFO  [ProcessThread(sid:3  cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34fcb982d030000 type:setData cxid:0x71 zxid:0x1000000d8 txntype:-1 reqpath:n/a Error Path:/config/topics/oneMoreTopic Error:KeeperErrorCode = NoNode for /config/topics/oneMoreTopic
2015-09-14 12:16:32,278 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34fcb982d030000 type:create cxid:0x72 zxid:0x1000000d9 txntype:-1 reqpath:n/a Error Path:/config/topics Error:KeeperErrorCode = NodeExists for /config/topics
2015-09-14 12:16:32,302 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34fcb982d030000 type:create cxid:0x7b zxid:0x1000000dc txntype:-1 reqpath:n/a Error Path:/brokers/topics/oneMoreTopic/partitions/0 Error:KeeperErrorCode = NoNode for /brokers/topics/oneMoreTopic/partitions/0
2015-09-14 12:16:32,304 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34fcb982d030000 type:create cxid:0x7c zxid:0x1000000dd txntype:-1 reqpath:n/a Error Path:/brokers/topics/oneMoreTopic/partitions Error:KeeperErrorCode = NoNode for /brokers/topics/oneMoreTopic/partitions

这似乎只是Zookeeper为该主题创建了一个新的Znode，因为它以前没有出现过。和Kafka server.log打印:

[2015-09-14 12:16:32,282] INFO Topic creation {"version":1,"partitions":{"0":[10200119]}} (kafka.admin.AdminUtils$)
[2015-09-14 12:16:32,287] INFO [KafkaApi-10200219] Auto creation of topic oneMoreTopic with 1 partitions and replication factor 1 is successful! (kafka.server.KafkaApis)
[2015-09-14 12:16:51,579] INFO Closing socket connection to /10.240.1.94. (kafka.network.Processor)

但是，我的消息从未发布到该主题，而下次我运行python-script时，我总是得到:

kafka.common.FailedPayloadsError

在我使其正常工作的情况下，advertised.host.name始终是节点的外部IP，但我似乎无法通过Kubernetes来实现。是否有可能使外部IP可从容器中调用？

我的kafka / config / server.properties对于所有代理来说都是这样的:

broker.id=10200121
host.name=kafka-f8p06
advertised.host.name=kafka-f8p06
++

Answer 1

broker.id=10200121
host.name=kafka-f8p06  <----- use IP here
advertised.host.name=kafka-f8p06  <---- use IP here

我认为您应该为 host.name和 advertised.host.name设置IP，因为K8不能通过主机名解析Pod，但是可以通过IP解析。

您的kafka节点可能无法以这种方式互相交谈，也找不到领导者。