hadoop - HMaster 在启动时被中止

Question

在 hadoop 2.2.0 的 hbase-0.96.0 中启动 ./start-hbase.sh 后，Hmaster 中止。

尝试使用 hbase-0.94.16 和 hbase-0.98 但结果相同。 Hmaster 一启动就中止。甚至尝试手动替换 hbase lib 中的 jars 以及使用 maven，但问题仍未解决。还有其他解决办法吗？

下面是对应的hbase-hadoop-master-hadoop-master.log...

2014-02-24 10:11:27,078 INFO  [Replication.RpcServer.handler=2,port=60000] ipc.RpcServer: Replication.RpcServer.handler=2,port=60000: starting
2014-02-24 10:11:27,565 INFO  [RpcServer.handler=23,port=60000] ipc.RpcServer: RpcServer.handler=23,port=60000: starting
2014-02-24 10:11:27,970 INFO  [master:hadoop-master:60000] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2014-02-24 10:11:28,172 INFO  [master:hadoop-master:60000] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2014-02-24 10:11:28,177 INFO  [master:hadoop-master:60000] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context master
2014-02-24 10:11:28,177 INFO  [master:hadoop-master:60000] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2014-02-24 10:11:28,191 INFO  [master:hadoop-master:60000] http.HttpServer: Jetty bound to port 60010
2014-02-24 10:11:28,191 INFO  [master:hadoop-master:60000] mortbay.log: jetty-6.1.26
2014-02-24 10:11:29,227 INFO  [master:hadoop-master:60000] mortbay.log: Started SelectChannelConnector@0.0.0.0:60010
2014-02-24 10:11:29,623 INFO  [master:hadoop-master:60000] master.ActiveMasterManager: Registered Active Master=hadoop-master.payoda.com,60000,1393236677609
2014-02-24 10:11:29,629 INFO  [master:hadoop-master:60000] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2014-02-24 10:11:29,851 DEBUG [main-EventThread] master.ActiveMasterManager: A master is now available
2014-02-24 10:11:30,537 INFO  [master:hadoop-master:60000] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-02-24 10:11:30,800 DEBUG [master:hadoop-master:60000] util.FSTableDescriptors: Current tableInfoPath = hdfs://hadoop-master:9000/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2014-02-24 10:11:30,821 DEBUG [master:hadoop-master:60000] util.FSTableDescriptors: TableInfo already exists.. Skipping creation
2014-02-24 10:11:30,944 INFO  [master:hadoop-master:60000] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2014-02-24 10:11:30,950 INFO  [master:hadoop-master:60000] master.SplitLogManager: Timeout=120000, unassigned timeout=180000, distributedLogReplay=false
2014-02-24 10:11:30,956 INFO  [master:hadoop-master:60000] master.SplitLogManager: Found 0 orphan tasks and 0 rescan nodes
2014-02-24 10:11:31,000 INFO  [master:hadoop-master:60000] zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.14.35:2181 sessionTimeout=90000 watcher=hconnection-0x4a867fad
2014-02-24 10:11:31,012 INFO  [master:hadoop-master:60000-SendThread(hadoop-master.payoda.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop-master.payoda.com/192.168.14.35:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
2014-02-24 10:11:31,617 INFO  [master:hadoop-master:60000-SendThread(hadoop-master.payoda.com:2181)] zookeeper.ClientCnxn: Socket connection established to hadoop-master.payoda.com/192.168.14.35:2181, initiating session
2014-02-24 10:11:31,617 INFO  [master:hadoop-master:60000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4a867fad connecting to ZooKeeper ensemble=192.168.14.35:2181
2014-02-24 10:11:31,620 INFO  [master:hadoop-master:60000-SendThread(hadoop-master.payoda.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server hadoop-master.payoda.com/192.168.14.35:2181, sessionid = 0x1446360aa4a0001, negotiated timeout = 90000
2014-02-24 10:11:31,640 DEBUG [master:hadoop-master:60000] catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3eaa3e5b
**2014-02-24 10:11:31,684 FATAL [master:hadoop-master:60000] master.HMaster: Unhandled exception. Starting shutdown.
java.lang.IllegalArgumentException: .META. no longer exists. The table has been renamed to hbase:meta**
    at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:292)
    at org.apache.hadoop.hbase.zookeeper.ZKTable.populateTableStates(ZKTable.java:82)
    at org.apache.hadoop.hbase.zookeeper.ZKTable.<init>(ZKTable.java:69)
    at org.apache.hadoop.hbase.master.AssignmentManager.<init>(AssignmentManager.java:281)
    at org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:677)
    at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:809)
    at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:603)
    at java.lang.Thread.run(Thread.java:662)
2014-02-24 10:11:31,684 INFO  [master:hadoop-master:60000] master.HMaster: Aborting
2014-02-24 10:11:31,711 DEBUG [master:hadoop-master:60000] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3eaa3e5b
2014-02-24 10:11:31,712 DEBUG [master:hadoop-master:60000] master.HMaster: Stopping service threads
2014-02-24 10:11:31,712 INFO  [master:hadoop-master:60000] ipc.RpcServer: Stopping server on 60000
2014-02-24 10:11:31,712 INFO  [RpcServer.handler=15,port=60000] ipc.RpcServer: RpcServer.handler=15,port=60000: exiting
2014-02-24 10:11:31,712 INFO  [RpcServer.handler=23,port=60000] ipc.RpcServer: RpcServer.handler=23,port=60000: exiting
2014-02-24 10:11:32,129 INFO  [master:hadoop-master:60000] master.HMaster: Stopping infoServer
2014-02-24 10:11:32,138 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2014-02-24 10:11:32,138 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2014-02-24 10:11:32,304 INFO  [hadoop-master.payoda.com,60000,1393236677609.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: hadoop-master.payoda.com,60000,1393236677609.splitLogManagerTimeoutMonitor exiting
2014-02-24 10:11:32,304 INFO  [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping
2014-02-24 10:11:32,304 INFO  [Replication.RpcServer.handler=2,port=60000] ipc.RpcServer: Replication.RpcServer.handler=2,port=60000: exiting
2014-02-24 10:11:32,304 INFO  [Replication.RpcServer.handler=1,port=60000] ipc.RpcServer: Replication.RpcServer.handler=1,port=60000: exiting
2014-02-24 10:11:32,304 INFO  [Replication.RpcServer.handler=0,port=60000] ipc.RpcServer: Replication.RpcServer.handler=0,port=60000: exiting
2014-02-24 10:11:32,304 INFO  [RpcServer.handler=29,port=60000] ipc.RpcServer: RpcServer.handler=29,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=28,port=60000] ipc.RpcServer: RpcServer.handler=28,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=27,port=60000] ipc.RpcServer: RpcServer.handler=27,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=26,port=60000] ipc.RpcServer: RpcServer.handler=26,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=25,port=60000] ipc.RpcServer: RpcServer.handler=25,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=24,port=60000] ipc.RpcServer: RpcServer.handler=24,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=22,port=60000] ipc.RpcServer: RpcServer.handler=22,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=21,port=60000] ipc.RpcServer: RpcServer.handler=21,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=20,port=60000] ipc.RpcServer: RpcServer.handler=20,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=19,port=60000] ipc.RpcServer: RpcServer.handler=19,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=18,port=60000] ipc.RpcServer: RpcServer.handler=18,port=60000: exiting
2014-02-24 10:11:32,305 INFO  [RpcServer.handler=17,port=60000] ipc.RpcServer: RpcServer.handler=17,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=16,port=60000] ipc.RpcServer: RpcServer.handler=16,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=14,port=60000] ipc.RpcServer: RpcServer.handler=14,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=13,port=60000] ipc.RpcServer: RpcServer.handler=13,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=12,port=60000] ipc.RpcServer: RpcServer.handler=12,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=11,port=60000] ipc.RpcServer: RpcServer.handler=11,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=10,port=60000] ipc.RpcServer: RpcServer.handler=10,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=9,port=60000] ipc.RpcServer: RpcServer.handler=9,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=8,port=60000] ipc.RpcServer: RpcServer.handler=8,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=7,port=60000] ipc.RpcServer: RpcServer.handler=7,port=60000: exiting
2014-02-24 10:11:32,306 INFO  [RpcServer.handler=6,port=60000] ipc.RpcServer: RpcServer.handler=6,port=60000: exiting
2014-02-24 10:11:32,307 INFO  [RpcServer.handler=5,port=60000] ipc.RpcServer: RpcServer.handler=5,port=60000: exiting
2014-02-24 10:11:32,307 INFO  [RpcServer.handler=4,port=60000] ipc.RpcServer: RpcServer.handler=4,port=60000: exiting
2014-02-24 10:11:32,307 INFO  [RpcServer.handler=3,port=60000] ipc.RpcServer: RpcServer.handler=3,port=60000: exiting
2014-02-24 10:11:32,307 INFO  [RpcServer.handler=2,port=60000] ipc.RpcServer: RpcServer.handler=2,port=60000: exiting
2014-02-24 10:11:32,307 INFO  [RpcServer.handler=1,port=60000] ipc.RpcServer: RpcServer.handler=1,port=60000: exiting
2014-02-24 10:11:32,307 INFO  [RpcServer.handler=0,port=60000] ipc.RpcServer: RpcServer.handler=0,port=60000: exiting
2014-02-24 10:11:32,930 INFO  [master:hadoop-master:60000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2014-02-24 10:11:32,945 INFO  [master:hadoop-master:60000] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1446360aa4a0001
2014-02-24 10:11:32,948 INFO  [master:hadoop-master:60000] zookeeper.ZooKeeper: Session: 0x1446360aa4a0001 closed
2014-02-24 10:11:32,949 INFO  [master:hadoop-master:60000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2014-02-24 10:11:32,954 INFO  [master:hadoop-master:60000] zookeeper.ZooKeeper: Session: 0x1446360aa4a0000 closed
2014-02-24 10:11:32,954 INFO  [master:hadoop-master:60000] master.HMaster: HMaster main thread exiting
2014-02-24 10:11:32,955 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2014-02-24 10:11:32,955 ERROR [main] master.HMasterCommandLine: Master exiting
**java.lang.RuntimeException: HMaster Aborted**
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:192)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2787)

Answer 1

您是否已从 HBase 0.94.x 升级到 0.96.x？从作为 RPC 机制迁移到 protobuffers 开始，它们就非常不兼容，是的，元表方法的变化。请确保您已检查升级文档。

http://hbase.apache.org/upgrading.html#upgrade0.96

请特别注意 ZooKeeper 服务。