hadoop - 无法在oozie中运行sqoop作业(没有枚举常量com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile)

Question

我成功创建并执行了sqoop导入作业，但是无法在oozie工作流程中运行它。 Sqoop将数据从RDBMS导入到HDFS中的 Parquet 文件中。看来问题与 Parquet 格式有关。如果我使用--as-textfile，则工作流运行不会出现任何问题。

另外，我已经将所有Parquet-*。jar从SQOOP_HOME / lib复制到oozie共享库。

Sqoop 1.4.7和Oozie 4.3.1。

Sqoop作业定义

$ sqoop job --create ingest_amsp_custmaster -- import --connect "jdbc:oracle:thin:@<IP>:<PORT>/<SID>" \
--username <USER> -P \
--table CUSTMASTER \
--as-parquetfile \
--target-dir /warehouse/raw/amsp/custmaster \
--delete-target-dir \
-m 1

这是我从错误日志中得到的结果:

java.lang.IllegalArgumentException: No enum constant com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile
        at java.lang.Enum.valueOf(Enum.java:238)
        at org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:522)
        at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:299)
        at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
        at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
        at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:200)
        at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:183)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:64)
        at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:436)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:350)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:211)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:254)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

任何帮助，将不胜感激。

谢谢。优s

Answer 1

在Oozie工作流程中检查Sqoop的版本。似乎不是1.4.7:堆栈跟踪行

org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:522)

对应于 https://github.com/apache/sqoop/blob/20af67ef60096b17e1d9585670e5ec787eb760e2/src/java/org/apache/sqoop/SqoopOptions.java#L522