hadoop - 如何从oozie执行sqoop工作？

Question

我无法使用sqoop命令从oozie执行示例作业，以将数据导入到配置单元中。我已经将hive-site.xml放在hdfs路径中，但是我认为它没有选择hive-site.xml文件。我正在上课找不到异常。如何解决这个问题？

工作流程

<!-- This is a comment -->
<workflow-app xmlns="uri:oozie:workflow:0.4" name="oozie-wf">
   <start to = "sqoop-node1" />
   <!--Step 1 -->
   <action name = "sqoop-node1" >
   <sqoop xmlns="uri:oozie:sqoop-action:0.2">
     <job-tracker></job-tracker>
     <name-node></name-node>
     <command> import command </command>
   </sqoop>
   <ok to="end"/>
   <error to="kill_job"/>
   </action>
   <kill name = "kill_job">
   <message>Job failed</message>
   </kill>
   <end name = "end" />
</workflow-app>

nameNode=ip jobTracker=ip queueName=default user.name=oozie oozie.use.system.libpath=true oozie.libpath=/user/hdfs/share/share/lib/sqoop oozie.wf.application.path=workflow path outputDir=/tmp/oozie.txt 

java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

Answer 1

我猜您的Sqoop操作需要 HCatalog库与Hive Metastore进行交互。而且Oozie默认情况下不会添加该库，您必须明确要求它。

请注意，有一些关于使用Pig的HCatalog的文献，但很少有Sqoop的文献。无论如何，窍门是一样的...
从Oozie documentation:

Oozie share libraries are organized per action type...
Oozie provides a mechanism to override the action share library JARs ...
More than one share library directory name can be specified for an action ... For example: When using HCatLoader and HCatStorer in pig, oozie.action.sharelib.for.pig can be set to pig,hcatalog to include both pig and hcatalog jars.

在您的情况下，您需要在Sqoop操作中使用值 <property>覆盖名为 oozie.action.sharelib.for.sqoop的特定 sqoop,hcatalog，然后Oozie将在运行时提供所需的JAR。