Hadoop:ClassNotFoundException - org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat

Question

当我为类 org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat 运行作业时，我遇到了 ClassNotFoundException。

我尝试使用 -libjars 传递额外的 jar 文件，但我仍然面临同样的问题。任何建议都会有很大帮助。提前致谢。

下面是我正在使用的命令和我面临的异常!

hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hcatalog/rcfile/RCFileMapReduceOutputFormat
at com.cloudera.sa.omniture.mr.OmnitureToRCFileJob.run(OmnitureToRCFileJob.java:91)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.cloudera.sa.omniture.mr.OmnitureToRCFileJob.main(OmnitureToRCFileJob.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 8 more

我也实现了 ToolRunner，下面是确认这一点的代码!

public class OmnitureToRCFileJob extends Configured implements Tool {

    public static void main(String[] args) throws Exception {

        OmnitureToRCFileJob processor = new OmnitureToRCFileJob();
        String[] otherArgs = new GenericOptionsParser(processor.getConf(), args).getRemainingArgs();

        System.exit(ToolRunner.run(processor.getConf(), processor, otherArgs));
    }

}

Answer 1

您是否尝试通过在下面的行中提供“hcatalog-core-0.5.0-cdh4.4.0.jar”jar 文件的完整路径来运行。

hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir

或

以下配置也适用于您

$ export LIBJARS= <fullpath>/hcatalog-core-0.5.0-cdh4.4.0.jar
$hadoop jar MyJob.jar MyDriver -libjars ${LIBJARS} inputDir OutputDir