hadoop - 避免在 MapReduce 作业输出文件的末尾附加 part-r-00*****

Question

我正在使用 Multioutputformat 类运行 MR 代码。 part**** 将附加在我的输出文件的末尾。我怎样才能避免这种情况？

公共(public)类 MR_reducer 扩展
reducer {

private MultipleOutputs multipleOutputs;

@Override
protected void setup(Context context) throws IOException,
        InterruptedException {
    multipleOutputs = new MultipleOutputs(context);
}


@Override
protected void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {
    for (Text value : values) {
        multipleOutputs.write(value, new Text(""), key.toString());
    }
}


@Override
protected void cleanup(Context context) throws IOException,
        InterruptedException {
    multipleOutputs.close();
}

}

Answer 1

这个代码片段对我有用。你有几个区别:

public static class Reduce extends Reducer<Text, Text, NullWritable, Text> {

    private MultipleOutputs<NullWritable, Text> multipleOutputs;

    protected void setup(Context context) throws IOException, InterruptedException {
        multipleOutputs = new MultipleOutputs<NullWritable, Text>(context);

    }

    public void reduce(Text key, Iterable<Text> values, Context output) throws IOException, InterruptedException {
        while (values.iterator().hasNext()) {
            multipleOutputs.write(NullWritable.get(), values.iterator().next(), key.toString());
        }
    }

    protected void cleanup(Context context) throws IOException, InterruptedException {
        multipleOutputs.close();
    }
}