hadoop - 使用自定义对象作为映射器发出的键

Question

我有一种情况，映射器将自定义类型的对象作为键发出。它有两个字段，一个 intWritable ID 和一个数据数组 IntArrayWritable。实现如下。`

import java.io.*;

import org.apache.hadoop.io.*;

public class PairDocIdPerm implements WritableComparable<PairDocIdPerm> {

    public PairDocIdPerm(){
        this.permId = new IntWritable(-1);
        this.SignaturePerm = new IntArrayWritable();
    }


public IntWritable getPermId() {
        return permId;
    }



    public void setPermId(IntWritable permId) {
        this.permId = permId;
    }



    public IntArrayWritable getSignaturePerm() {
        return SignaturePerm;
    }



    public void setSignaturePerm(IntArrayWritable signaturePerm) {
        SignaturePerm = signaturePerm;
    }

    private IntWritable permId;
    private IntArrayWritable SignaturePerm;

    public PairDocIdPerm(IntWritable permId,IntArrayWritable SignaturePerm) {
   this.permId = permId;
   this.SignaturePerm = SignaturePerm;
   }



    @Override
    public void write(DataOutput out) throws IOException {
    permId.write(out);
    SignaturePerm.write(out);
    }

   @Override
    public void readFields(DataInput in) throws IOException {
    permId.readFields(in);
    SignaturePerm.readFields(in);
    }

    @Override
    public int hashCode() { // same permId must go to same reducer. there fore just permId
     return permId.get();//.hashCode(); 
    }

    @Override
    public boolean equals(Object o) {
     if (o instanceof PairDocIdPerm) {
   PairDocIdPerm tp = (PairDocIdPerm) o;
   return permId.equals(tp.permId) && SignaturePerm.equals(tp.SignaturePerm);
     }
     return false;
    }

    @Override
    public String toString() {
     return permId + "\t" +SignaturePerm.toString(); 
    }

    @Override
    public int compareTo(PairDocIdPerm tp) {
     int cmp = permId.compareTo(tp.permId);
     Writable[] ar, other;
     ar = this.SignaturePerm.get();
     other = tp.SignaturePerm.get();

    if (cmp == 0) {
     for(int i=0;i<ar.length;i++){
         if(((IntWritable)ar[i]).get() == ((IntWritable)other[i]).get()){cmp= 0;continue;}
        else if(((IntWritable)ar[i]).get() < ((IntWritable)other[i]).get()){ return -1;}
         else if(((IntWritable)ar[i]).get() > ((IntWritable)other[i]).get()){return 1;}
     }   
     }

     return cmp;
     //return 1;
   }

   }`

我要求具有相同 Id 的键按照 compareTo 方法中编码的排序顺序进入同一个 reducer。但是，当我使用它时，我的作业执行状态始终是 map100% reduce 0%。reduce 永远不会运行完成。这个实现有什么问题吗？一般来说，如果 reducer 状态始终为 0%，可能会出现什么问题。

Answer 1

我认为这可能是读取方法中的空指针异常:

   @Override
    public void readFields(DataInput in) throws IOException {
    permId.readFields(in);
    SignaturePerm.readFields(in);
    }

permId 在这种情况下为空。所以你要做的是:

IntWritable permId = new IntWritable();

在字段初始值设定项中或读取之前。

但是，您的代码很难读。