hadoop - Hbase批量加载失败| KosmosFileSystem文件系统异常

Question

我刚刚提到以下博客来运行简单的大容量加载，但是它给了KosmosFileSystem FileSystem异常。

http://www.thecloudavenue.com/2013/04/bulk-loading-data-in-hbase.html

这是运行以下命令后生成的日志。

hadoop jar HbaseBulkImport.jar /user/hduser/hbase/input/RowFeeder.csv / user / hduser / hbase / ouput / NBAFinal2010

15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.2.0.0-2041/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.2.0.0-2041/hadoop/lib/native
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-504.12.2.el6.x86_64
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:user.name=hduser
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hduser/user/shashi/hbase/bulkLoad
15/04/16 12:40:35 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x2dd06f21, quorum=localhost:2181, baseZNode=/hbase-unsecure
15/04/16 12:40:35 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2dd06f21 connecting to ZooKeeper ensemble=localhost:2181
15/04/16 12:40:35 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
15/04/16 12:40:35 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
15/04/16 12:40:35 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x34cb73f98050030, negotiated timeout = 40000
Exception in thread "main" java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:426)
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:403)
        at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:281)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:207)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:169)
        at Driver.main(Driver.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:424)
        ... 11 more
Caused by: java.lang.ExceptionInInitializerError
        at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
        at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:106)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:858)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:662)
        ... 16 more
Caused by: java.lang.UnsupportedOperationException: Not implemented by the KosmosFileSystem FileSystem implementation
        at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:104)
        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:226)
        ... 21 more

Answer 1

看起来是与jar相关的问题。从hbase lib使用最新的jar后，问题得到解决。