hadoop - Sqoop 导出失败 - 找不到 ExportBatchOutputFormat

Question

当我尝试将 hive 分区导出到我的 Oracle 数据库时，导出失败，我无法弄清楚为什么......有人可以帮我吗？

我的甲骨文表:

  CREATE TABLE "GFPROGNOSE"."HADOOP_TEST" 
   ("COLUMN1" VARCHAR2(255), 
    "COLUMN2" VARCHAR2(255), 
    "COLUMN3" VARCHAR2(255), 
    "COLUMN4" VARCHAR2(255), 
    "DS_YEAR" VARCHAR2(255), 
    "DS_MONTH" VARCHAR2(255)
   )

我的 hive 表:
将 Oracle 表的所有列作为字符串和其他列

我的独家新闻导出命令:

 sudo -u hdfs sqoop-export --connect jdbc:oracle:thin:@192.168.145.210:1521:dmine
 --table HADOOP_TEST --username USERNAME --password xxxxxx --num-mappers 1 
--update-mode allowinsert --skip-dist-cache --hcatalog-database default 
--hcatalog-table adobe_archive --hcatalog-partition-keys ds_year,ds_month,ds_day
 --hcatalog-partition-values 2017,06,01

命令行输出:

17/08/10 10:03:56 INFO mapreduce.Job:  map 0% reduce 0%
17/08/10 10:03:56 INFO mapreduce.Job: Job job_1501840909803_1851 failed with state FAILED due to: Application application_1501840909803_1851 failed 2 times due to AM Container for appattempt_1501840909803_1851_000002 exited with  exitCode: 1
For more detailed output, check the application tracking page: http://master.joinplus.local:8088/cluster/app/application_1501840909803_1851 Then click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e17_1501840909803_1851_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:944)
        at org.apache.hadoop.util.Shell.run(Shell.java:848)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1142)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

申请日志:

2017-08-10 10:03:55,839 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2017-08-10 10:03:55,880 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.cloudera.sqoop.mapreduce.ExportBatchOutputFormat not found
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.cloudera.sqoop.mapreduce.ExportBatchOutputFormat not found
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:520)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:500)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1601)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:500)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:286)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1559)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1556)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1487)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.cloudera.sqoop.mapreduce.ExportBatchOutputFormat not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2241)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:516)
        ... 11 more
Caused by: java.lang.ClassNotFoundException: Class com.cloudera.sqoop.mapreduce.ExportBatchOutputFormat not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2239)
        ... 13 more

Answer 1

现在它起作用了。
问题是 --skip-dist-cache sqoop 命令中的参数。
它在那里是因为我从 falcon 为调试目的生成的 oozie 工作流中复制了命令。