apache-storm - 什么会导致zookeeper Client session超时

Question

我部署了一个长期运行的 Storm 拓扑。运行几个小时后，整个拓扑出现故障。我检查了 worker 日志，并找到了这些日志。正如它所说，zookeeper 客户端 session 超时并导致重新连接。我怀疑这与我损坏的拓扑有关。现在我尝试找出可能导致客户端超时的原因。

2016-02-29T10:34:12.386+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 23789ms for sessionid 0x252f862028c0083, closing socket connection and attempting reconnect
2016-02-29T10:34:12.986+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED
2016-02-29T10:34:13.059+0800 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper.
2016-02-29T10:34:13.197+0800 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zk-3.cloud.mos/172.16.13.147:2181. Will not attempt to authenticate using SASL (unknown error)
2016-02-29T10:34:13.241+0800 o.a.s.z.ClientCnxn [WARN] Session 0x252f862028c0083 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_31]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_31]
    at org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[storm-core-0.9.6.jar:0.9.6]
    at org.apache.storm.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[storm-core-0.9.6.jar:0.9.6]

Answer 1

您的客户端无法再与 ZooKeeper 服务器通信。发生的第一件事是在协商的 session 超时内没有心跳响应:

2016-02-29T10:34:12.386+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 23789ms for sessionid 0x252f862028c0083, closing socket connection and attempting reconnect

然后当它尝试重新连接时，连接被拒绝:

2016-02-29T10:34:13.241+0800 o.a.s.z.ClientCnxn [WARN] Session 0x252f862028c0083 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused

这意味着您的 ZooKeeper 服务器:

无法访问(网络连接中断)

已死(所以没有监听套接字)

GC 本身是否已死且无法通信(尽管这可能会发出连接超时错误，但我不确定)

要了解更多信息，您需要检查(Hadoop？)集群上的 ZooKeeper 服务器日志。