java - Hadoop - 如何发射 TwoDArrayWritable

Question

我正在从映射器发出 TwoDArrayWritable。我已经实现了扩展 TwoDArrayWritable 的类，以便创建默认构造函数。但是当我尝试发出它时，它给了我以下异常:

Error: java.lang.RuntimeException: java.lang.InstantiationException: org.apache.hadoop.io.TwoDArrayWritable
        at org.apache.hadoop.io.TwoDArrayWritable.readFields(TwoDArrayWritable.java:75)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)

我如何发射 TwoDArrayWritable？我需要一些帮助。

这是扩展 TwoDArrayWritable 的类:

public class TwoDArrayWritables extends TwoDArrayWritable
{
    public TwoDArrayWritables() {
        super(TwoDArrayWritable.class);

    }


    public TwoDArrayWritables(Class valueClass) {
        super(valueClass);
        // TODO Auto-generated constructor stub
    }

这是映射器:

public class JaccardMapper extends Mapper<LongWritable, Text, IntTextPair, TwoDArrayWritables> {

    Hashtable movieInfo = new Hashtable<String, String>();
    String[] genres, actors, entities;
    String[] attributes = new String[] {"genre", "actors", "directors", "country", "year", "ratings"};
    double p,q,r,s;
    double result = 0.0;
    String input[] = null;
    Set<String> keys;

    TwoDArrayWritables array2d = new TwoDArrayWritables();
    //TwoDArrayWritable array2d = new TwoDArrayWritable(IntWritable.class);

    IntWritable[][] jaccard = new IntWritable[2][];
    //int[][] jaccard = new int[2][];


    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException
    {

        p = 0;
        q = 0;
        r = 0;
        s = 0;

        input = value.toString().toLowerCase().split(",");
        keys = movieInfo.keySet();


        //the jaccards 2d array column length depends on the user input best case is 6 but the worst case depends on the sub attributes count like more than one actor/director/genre/country.  
        int columnlength = input[1].split("\\|").length + input[2].split("\\|").length + input[3].split("\\|").length + input[4].split("\\|").length + 2;
        jaccard = new IntWritable[2][columnlength];
        for (int i = 0; i < jaccard.length; i++)
        {
            for (int j = 0; j < jaccard[i].length; j++)
            {
                jaccard[i][j] = new IntWritable(0);
            }
        }

        if (input.length > 0)
        {
            //iterate through the dataset in cache
            for(String keyy : keys)
            {
                //iterate to user's input attributes
                for (int attribute = 1; attribute < attributes.length; attribute++)
                {
                    if (!input[attribute].equals("-")) 
                    {
                        entities = input[attribute].toLowerCase().split("\\|");
                        int subattributecount = 0;

                        for(String entity : entities)
                        {

                                subattributecount += 1;
                        }
                    }
                }
                IntTextPair pair = new IntTextPair(Integer.parseInt(input[0].toString()), movieInfo.get(keyy).toString());

                array2d.set(jaccard);
                                context.write(pair, array2d);
            }


        }

}
}

这是 reducer :公共(public)类 JaccardReducer 扩展 Reducer {

double p,q,r,s;
double result = 0.0;
//IntWritable[][] jaccard = null;
IntWritable[][] jaccard;
int temp1 = 0, temp2 = 0;

//Jaccard distance 1.0 emplies that the user's criteria doesn't meet at all
public static final double nonacceptvalue = 1.0;


    public void reduce(IntTextPair key, Iterable<TwoDArrayWritables> values, Context context) throws IOException, InterruptedException
    {

        IntDoublePair pair = new IntDoublePair(key.getFirst().get(), Double.parseDouble("110.00"));
        for (TwoDArrayWritable value : values)
        {
            if (value != null)
            {
                context.write(pair, new Text("Is not null"));
            }
            else
            {
                context.write(pair, new Text("Is null"));
            }
        }
}
}

Answer 1

您在默认构造函数(在 super 中)中指定的类应该是值的类，而不是整个类本身。所以你可能想要:

public TwoDArrayWritables() {
    super(IntWritable.class);

}