hadoop - 将一个MapReduce的输出用作另一个MapReduce的输入时出错

Question

我有两个Map / Reduce类，分别名为MyMappper1 / MyReducer1和MyMapper2 / MyReducer2，并希望通过将job2的输入路径设置为job1的输出路径，将MyReducer1的输出用作MyMapper2的输入。

类型如下:

    public class MyMapper1 extends Mapper<LongWritable, Text, IntWritable, IntArrayWritable>
    public class MyReducer1 extends Reducer<IntWritable, IntArrayWritable, IntWritable, IntArrayWritable>
    public class MyMapper2 extends Mapper<IntWritable, IntArrayWritable, IntWritable, IntArrayWritable>
    public class MyReducer2 extends Reducer<IntWritable, IntArrayWritable, IntWritable, IntWritable>

public class IntArrayWritable extends ArrayWritable {
    public IntArrayWritable() {
        super(IntWritable.class);
    }
}

设置输入/输出路径的代码如下:

    Path temppath = new Path("temp-dir-" + temp_time);

    FileOutputFormat.setOutputPath(job1, temppath);

            ...........

    FileInputFormat.addInputPath(job2, temppath);

设置输入/输出格式的代码如下:

    job1.setOutputFormatClass(TextOutputFormat.class);
            ..........
    job2.setInputFormatClass(KeyValueTextInputFormat.class);

但是，在运行job2时，我总是会遇到异常:

11/04/16 12:34:09 WARN mapred.LocalJobRunner: job_local_0002
java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable
    at ligon.MyMapper2.map(MyMapper2.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

我尝试更改InputFormat和OutputFormat，但没有成功，在job2中发生了类似(尽管不相同)异常。

我完整的代码包位于:
http://dl.dropbox.com/u/7361939/HW2_Q1.zip

Answer 1

问题在于，在作业2中，KeyValueTextInputFormat生成类型为key-value的键-值对，并且您尝试使用具有accepts的Mapper处理它们，从而导致ClassCastException。最好的选择是将您的映射器更改为接受并从Text转换为integer。