hadoop - Hbase Pig错误

Question

我有以MAPREDUCE模式运行的Hadoop 2.6.1，PIG 0.15.0。 HBase 1.1.1
当我尝试执行
raw =使用org.apache.pig.backend.hadoop.hbase.HBaseStorage('test_cf2:NAME')AS(NAME:chararray)加载'hbase:// test_table2';

我收到以下错误
[hadoop @ hdnn脚本] $ pig sample1.pig

2015-11-18 23:52:14,667 INFO  [main] pig.ExecTypeProvider: Trying ExecType : LOCAL
2015-11-18 23:52:14,668 INFO  [main] pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
2015-11-18 23:52:14,668 INFO  [main] pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-11-18 23:52:14,718 [main] INFO  org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
2015-11-18 23:52:14,718 [main] INFO  org.apache.pig.Main - Logging error messages to: /usr/lib/pig/pig-0.15.0/scripts/pig_1447870934717.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-11-18 23:52:15,202 [main] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-11-18 23:52:15,943 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found
2015-11-18 23:52:16,381 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-11-18 23:52:16,381 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-11-18 23:52:16,381 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://hdnn:9001
2015-11-18 23:52:18,020 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf2:NAME]'
Failed to parse: Pig script failed to parse:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf2:NAME]'
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
    at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735)
    at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443)
    at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:412)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
    at org.apache.pig.Main.run(Main.java:631)
    at org.apache.pig.Main.main(Main.java:177)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf2:NAME]'
    at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897)
    at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
    at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
    at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
    at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
    at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
    ... 16 more
Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf2:NAME]'
    at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772)
    at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885)
    ... 22 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740)
    ... 23 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V
    at org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan(HBaseStorage.java:427)
    at org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:368)
    at org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:239)
    ... 28 more
2015-11-18 23:52:18,025 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse:
<file sample1.pig, line 1, column 6> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[test_cf2:NAME]'

我复制hbase-0.94.27.jar后，上述错误消失了，但又收到另一个错误，如下
[hadoop @ hdnn脚本] $ pig sample1.pig

2015-11-19 00:19:48,353 INFO  [main] pig.ExecTypeProvider: Trying ExecType : LOCAL
2015-11-19 00:19:48,353 INFO  [main] pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
2015-11-19 00:19:48,353 INFO  [main] pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-11-19 00:19:48,448 [main] INFO  org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
2015-11-19 00:19:48,448 [main] INFO  org.apache.pig.Main - Logging error messages to: /usr/lib/pig/pig-0.15.0/scripts/pig_1447872588445.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-11-19 00:19:49,012 [main] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-11-19 00:19:49,690 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found
2015-11-19 00:19:50,120 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-11-19 00:19:50,120 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-11-19 00:19:50,120 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://hdnn:9001
2015-11-19 00:19:51,770 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. Bad type on operand stack in method org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan()V at offset 320
Details at logfile: /usr/lib/pig/pig-0.15.0/scripts/pig_1447872588445.log
2015-11-19 00:19:51,816 [main] INFO  org.apache.pig.Main - Pig script completed in 3 seconds and 535 milliseconds (3535 ms)

我尝试在本地计算机上为PIG构建ant。构建了piggybank.jar，但没有任何工作。
如果HBaseStorage出现问题，如何处理HBase中的表数据。

请帮忙。

Answer 1

这是由于Pig 15和hbase 112之间存在兼容性问题。
https://issues.apache.org/jira/browse/PIG-4728

下载Pig 15源代码并使用以下命令对其进行编译

Ant 清理-Dhadoopversion = 23 -Dhbase95.version = 1.1.1 jar

这会起作用。