java - 实现树可写类

Question

我想实现一个 TreeWritable 类来表示一个树结构。我尝试了以下实现，但出现了 mapred.MapTask: Record too large for in-memory buffer 错误。我应该如何为多级数据结构实现 Writable？

public class TreeWritable implements Writable
{
private final Set<TreeWritable> children = new LinkedHashSet<TreeWritable>();
private String data;
private int level;

public TreeWritable( String data, int level )
{
    this.data = data;
    this.level = level;
}

public int getLevel()
{
    return level;
}

public TreeWritable()
{
}

public TreeWritable child( String data )
{
    for ( TreeWritable child : children )
    {
        if ( child.data.equals( data ) )
        {
            return child;
        }
    }
    return child( new TreeWritable( data, this.level + 1 ) );
}

TreeWritable child( TreeWritable child )
{
    children.add( child );
    return child;
}

public Set<TreeWritable> getChildren()
{
    return children;
}

public String getId()
{
    return data;
}

public void write( DataOutput out ) throws IOException
{
    out.writeUTF( data );
    out.write( level );
    int size = children.size();
    out.writeInt( size );        
    while(children.iterator().hasNext())
        children.iterator().next().write( out );
}

public void readFields( DataInput in ) throws IOException
{
    data = in.readUTF();
    level = in.readInt();
    int size = in.readInt();
    for ( int i = 0; i < size; i++ )
        children.add( TreeWritable.read( in ) );
}

public static TreeWritable read( DataInput in ) throws IOException
{
    TreeWritable w = new TreeWritable();
    w.readFields( in );
    return w;
}
}

Answer 1

我认为如果要在单机上处理它是小树的最佳实现。如果你正在处理大树，你应该将它拆分成树的部分，并将它存储为元组(例如，id、data、root_id)。

另一个例子，MapReduce中PageRank评估的数据结构是(url, currentPageRank, [link_url1, link_url2, ...])