hadoop - 使用 Oozie 将 hdfs 文件映射到 HBase

Question

我正在尝试从 Oozie 启动一个 shell 作业，我将文件从 hdfs 映射到 Hbase

Shell 脚本在命令行下工作，但在 Oozie 下不工作

外壳脚本:

HADOOP_CLASSPATH=`/usr/bin/hbase classpath` /usr/bin/hadoop jar /usr/lib/hbase/hbase-server.jar importtsv -Dimporttsv.separator=, -Dimporttsv.columns=ORIGINAL:FIELD1,ORIGINAL:FIELD2,ORIGINAL:FIELD3  HBASE_Table /user/U0H8048/file  

错误是:

SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/cloudera/opt/cloudera/parcels/CDH-5.0.2-    1.cdh5.0.2.p0.13/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/cloudera/yarn/nm/filecache/2282/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/cloudera/yarn/nm/filecache/2288/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
  Caused by: java.io.FileNotFoundException: File does not exist: hdfs://salcls0226.com:8020/cloudera/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hbase/lib/hbase-server-0.96.1.1-cdh5.0.2.jar
  at    org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
  at    org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
      ...

有谁知道可能的解决方案吗？

如何指定 hbase-server-0.96.1.1-cdh5.0.2.jar 文件位于本地路径而不是 hdfs 上？

谢谢

Answer 1

我解决了这个问题。

它需要在启动工作流时将 oozie.use.system.libpath=true 行添加到 Oozie 读取的 job.properties 文件中。

您必须将所有需要的文件添加到 HDFS 路径:/user/oozie/share/lib。通常主库文件已经存在。