hadoop - HMaster 启动后很快停止

Question

我看过很多类似的问题并尝试解决但尚未成功。我有一个 hadoop 2.6.0 集群(一个名称节点位于 x.x.x.55，3 个数据节点位于 x.x.x.54、56、57)。然后我在集群上安装了 hbase-0.98.12-hadoop2。在 hbase-env.sh 中，我取消注释 HBASE_MANAGES_ZK = true，并设置 Java 主路径。这是 hbase-site.xml:

<property>
    <name>hbase.master</name>
    <value>hadoop.master:60000</value>
</property>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hadoop.master:9000/hbase</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2222</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>hadoop.master</value>
</property>

刚启动hbase，jps就打印出来了:

5960 NameNode
7635 HMaster
8062 Jps
6154 SecondaryNameNode
7537 HQuorumPeer

但很快(1 或 2 分钟)，HMaster 就停止了。这是 Hmaster 日志:

2015-06-22 15:47:13,286 INFO  [master:hadoop:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 856 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-06-22 15:47:14,803 INFO  [master:hadoop:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 2373 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-06-22 15:47:16,315 INFO  [master:hadoop:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 3, slept for 3885 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-06-22 15:47:16,974 INFO  [master:hadoop:60000] master.ServerManager: Finished waiting for region servers count to settle; checked in 3, slept for 4544 ms, expecting minimum of 1, maximum of 2147483647, master is running.
2015-06-22 15:47:16,994 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.master,60020,1434939246333 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,994 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,16020,1434708308676-splitting doesn't belong to a known region server, splitting
2015-06-22 15:47:16,995 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,16020,1434935680372 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,995 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434939607069 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434940811464 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434941688620 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434950660701 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434952134591 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,996 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave1,60020,1434955629747 belongs to an existing region server
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,16020,1434708308695-splitting doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,16020,1434935679919 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434939606935 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434940810938 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434941688163 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434950660936 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,997 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434952134902 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,998 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave2,60020,1434955629669 belongs to an existing region server
2015-06-22 15:47:16,998 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,16020,1434935540949 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,998 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,16020,1434935680550-splitting doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434939607523 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434940811989 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434941689997 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434950660287 doesn't belong to a known region server, splitting
2015-06-22 15:47:16,999 INFO  [master:hadoop:60000] master.MasterFileSystem: Log folder hdfs://hadoop.master:9000/hbase/WALs/hadoop.slave3,60020,1434955629745 belongs to an existing region server
2015-06-22 15:47:17,000 FATAL [master:hadoop:60000] master.HMaster: Master server abort: loaded coprocessors are: []
2015-06-22 15:47:17,002 FATAL [master:hadoop:60000] master.HMaster: Unhandled exception. Starting shutdown.
java.lang.IllegalArgumentException: Not a host:port pair: hregion-79918053
at org.apache.hadoop.hbase.util.Addressing.parseHostname(Addressing.java:62)
at org.apache.hadoop.hbase.ServerName.<init>(ServerName.java:128)
at org.apache.hadoop.hbase.ServerName.valueOf(ServerName.java:177)
at org.apache.hadoop.hbase.ServerName.parseServerName(ServerName.java:347)
at org.apache.hadoop.hbase.master.MasterFileSystem.getFailedServersFromLogFolders(MasterFileSystem.java:253)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:944)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:683)
at java.lang.Thread.run(Thread.java:745)
2015-06-22 15:47:17,006 INFO  [master:hadoop:60000] master.HMaster: Aborting
2015-06-22 15:47:17,007 DEBUG [master:hadoop:60000] master.HMaster: Stopping service threads
2015-06-22 15:47:17,007 INFO  [master:hadoop:60000] ipc.RpcServer: Stopping server on 60000
2015-06-22 15:47:17,007 INFO  [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping
2015-06-22 15:47:17,010 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2015-06-22 15:47:17,011 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2015-06-22 15:47:17,013 INFO  [master:hadoop:60000.oldLogCleaner] cleaner.LogCleaner: master:hadoop:60000.oldLogCleaner exiting
2015-06-22 15:47:17,013 INFO  [master:hadoop:60000.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x14e1a05351e0005, quorum=hadoop.master:2222, baseZNode=/hbase
2015-06-22 15:47:17,014 INFO  [master:hadoop:60000.archivedHFileCleaner] cleaner.HFileCleaner: master:hadoop:60000.archivedHFileCleaner exiting
2015-06-22 15:47:17,016 INFO  [master:hadoop:60000] master.HMaster: Stopping infoServer
2015-06-22 15:47:17,017 INFO  [master:hadoop:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-22 15:47:17,017 INFO  [master:hadoop:60000.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x14e1a05351e0005 closed
2015-06-22 15:47:17,024 INFO  [master:hadoop:60000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2015-06-22 15:47:17,031 DEBUG [master:hadoop:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@71e15889
2015-06-22 15:47:17,031 INFO  [master:hadoop:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14e1a05351e0004
2015-06-22 15:47:17,034 INFO  [master:hadoop:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-22 15:47:17,034 INFO  [master:hadoop:60000] zookeeper.ZooKeeper: Session: 0x14e1a05351e0004 closed
2015-06-22 15:47:17,034 INFO  [hadoop.master,60000,1434955628046.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: hadoop.master,60000,1434955628046.splitLogManagerTimeoutMonitor exiting
2015-06-22 15:47:17,037 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-22 15:47:17,037 INFO  [master:hadoop:60000] zookeeper.ZooKeeper: Session: 0x14e1a05351e0000 closed
2015-06-22 15:47:17,037 INFO  [master:hadoop:60000] master.HMaster: HMaster main thread exiting
2015-06-22 15:47:17,038 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:201)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3029)

注意:所有节点使用相同的hbase-0.98.12。

请帮助解决这个问题。

Answer 1

You have java.lang.IllegalArgumentException: Not a host:port pair: exception. This arises mostly due to incompatible versions of hbase with other componenets.

You can have a look at this compatibility matrix, your hadoop 2.6.0 cluster with hbase 0.98 is not tested.

http://hbase.apache.org/book.html#hadoop