java - 复杂类型的缩减器中 ReadFields() 中的空缓冲区

Question

我正在尝试在 mapper 和 reducer 之间传递一个复杂的可写对象，更具体地说是 ObjectWritables 的 ArrayWritable。

public class ObjectArrayWritable extends ArrayWritable {
    public ObjectArrayWritable() {
        super(ObjectWritable.class);
    }
}

我的映射器:Mapper<LongWritable, Text, Text, ObjectArrayWritable>

我的 reducer :Reducer<Text,ObjectArrayWritable,Text,ObjectArrayWritable>

reducer 在 readfields() 上崩溃可写接口(interface)的方法并抛出以下异常:

java.lang.NullPointerException
    at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:183)
    at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
    at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)

当我尝试实现自己的自定义可写并对其进行调试时，我注意到 readfields(DataInput data) 中的 DataInput 缓冲区收到空..

请注意，我的错误类似于: hadoop + Writable interface + readFields throws an exception in reducer但是清洁物体并没有帮助..

有人可以帮忙吗？

Answer 1

我不认为您正在处理 ObjectArrayWritable 中没有任何内容的情况。我不能确定，因为没有提供代码。