hadoop - Avro Map Reduce-未找到AvroInputFormat错误

Question

到目前为止，我从互联网上的各种来源中了解到这一点。

Avro mapred and Avro are not part of CDH4 (Cloudera Distribution) and i have to set it up manually using HADOOP_CLASSPATH=avro.jar:avro-mapred.jar

我已经做到了，当我在我的伪集群上运行我的工作时，它抛出以下异常:

13/12/27 00:47:40 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

13/12/27 00:47:40 INFO mapred.FileInputFormat: Total input paths to process : 1

13/12/27 00:47:41 INFO mapred.JobClient: Running job: job_201312221245_0017

13/12/27 00:47:42 INFO mapred.JobClient: map 0% reduce 0%

13/12/27 00:47:57 INFO mapred.JobClient: Task Id : attempt_201312221245_0017_m_000000_0, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.avro.mapred.AvroInputFormat not found

我正在按以下方式运行作业:

hadoop jar build/libs/hadoop-boilerplate-1.0.jar CustomerMapReduce transactions/input transactions/output1 -libjars /path/to/libs/avro-1.7.4.jar,/path/to/libs/avro-mapred-1.7.4.jar

Answer 1

您应该实现工具，并使用 getConf()进行作业配置。

public class SomeClass extends Configured implements Tool {
    @Override
    public int run(String[] args) throws Exception {
        Configuration conf = getConf();
        ...
    }
}