hadoop - 无法在 Cloudera Manager 中启动 HBase

Question

在我的设置中，由于电源中断，我的虚拟机在重新启动所有虚拟机后进入停机状态我无法启动 Hbase，所有其他服务都可以重新启动/停止/启动，但在 CM 的 UI 中在 hbase ACTIONS 选项卡中，所有选项都被隐藏了..

以下是我的hbase master日志，现在处于退出模式

6:29:05.880 AM  ERROR   org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
ZooKeeper exists failed after 3 retries
6:29:05.880 AM  WARN    org.apache.hadoop.hbase.zookeeper.ZKUtil    
hconnection Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:420)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:621)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:684)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:126)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.stopMaster(HMasterCommandLine.java:175)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:106)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1962)
6:29:05.880 AM  ERROR   org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher  
hconnection Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:420)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:621)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:684)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:126)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.stopMaster(HMasterCommandLine.java:175)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:106)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1962)
6:29:05.881 AM  INFO    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation 
This client just lost it's session with ZooKeeper, will automatically reconnect when needed.
6:29:06.880 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderaalpha.alp.com/10.0.0.13:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:06.882 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:06.983 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:29:06.983 AM  INFO    org.apache.hadoop.hbase.util.RetryCounter   
Sleeping 2000ms before retry #1...
6:29:06.983 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn1.alp.com/10.0.0.14:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:06.984 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:07.084 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn3.alp.com/10.0.0.16:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:27.103 AM  INFO    org.apache.zookeeper.ClientCnxn 
Client session timed out, have not heard from server in 20019ms for sessionid 0x0, closing socket connection and attempting reconnect
6:29:27.204 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:29:27.204 AM  INFO    org.apache.hadoop.hbase.util.RetryCounter   
Sleeping 4000ms before retry #2...
6:29:28.204 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderaalpha.alp.com/10.0.0.13:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:28.206 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:28.306 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn1.alp.com/10.0.0.14:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:28.307 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:28.408 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn3.alp.com/10.0.0.16:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:48.425 AM  INFO    org.apache.zookeeper.ClientCnxn 
Client session timed out, have not heard from server in 20017ms for sessionid 0x0, closing socket connection and attempting reconnect
6:29:48.526 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:29:48.526 AM  INFO    org.apache.hadoop.hbase.util.RetryCounter   
Sleeping 8000ms before retry #3...
6:29:49.526 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderaalpha.alp.com/10.0.0.13:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:49.529 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:49.630 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn1.alp.com/10.0.0.14:2181. Will not attempt to authenticate using SASL (unknown error)
6:29:49.630 AM  WARN    org.apache.zookeeper.ClientCnxn 
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
6:29:49.731 AM  INFO    org.apache.zookeeper.ClientCnxn 
Opening socket connection to server cderadn3.alp.com/10.0.0.16:2181. Will not attempt to authenticate using SASL (unknown error)
6:30:09.744 AM  INFO    org.apache.zookeeper.ClientCnxn 
Client session timed out, have not heard from server in 20013ms for sessionid 0x0, closing socket connection and attempting reconnect
6:30:09.845 AM  WARN    org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
6:30:09.845 AM  ERROR   org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  
ZooKeeper exists failed after 3 retries

感谢您的关注...

Answer 1

从日志中我们可以看到连接ZooKeeper服务器失败。您可能需要启动 ZooKeeper 服务并尝试重启 HBase