hadoop - 如何维护 Reducer 中 MapWritables 的顺序？

Question

我的映射器实现

public class SimpleMapper extends Mapper<Text, Text, Text, MapWritable> {

@Override
protected void map(Text key, Text value,Context context)
        throws IOException, InterruptedException {

            MapWritable writable = new LinkedMapWritable();
            writable.put("unique_key","one");
            writable.put("another_key","two");
            context.write(new Text("key"),writable );
        }

Reducer 实现是:

public class SimpleReducer extends Reducer<Text, MapWritable, NullWritable, Text> {
@Override
protected void reduce(Text key, Iterable<MapWritable> values,Context context)
        throws IOException, InterruptedException {

            // The map writables have to be ordered based on the "unique_key" inserted into it
        }

我必须使用二级排序吗？还有其他方法吗？

Answer 1

reducer 中的 MapWritable (values) 始终处于不可预测的顺序，此顺序可能会因运行而异，您无法控制它。

但是 Map/Reduce 范式保证的是呈现给 reducer 的键将按排序顺序排列，属于单个键的所有值将进入单个 reducer。

因此，您绝对可以为您的用例使用辅助排序和自定义分区程序。