hadoop - 通过分布式缓存访问Mapper中的文件

Question

我想访问Mapper中的分布式文件的内容。下面是我编写的代码，该代码生成分布式缓存的文件名。请帮助我访问文件内容

   public class DistCacheExampleMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text >
     {
      Text a = new Text();
    Path[] dates = new Path[0];
    public void configure(JobConf conf) {

    try {
            dates = DistributedCache.getLocalCacheFiles(conf);
            String astr = dates.toString();
            a = new Text(astr);

          } catch (IOException ioe) {
            System.err.println("Caught exception while getting cached files: " +   
          StringUtils.stringifyException(ioe));
          }


    }

    @Override
    public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, 
           Reporter reporter) throws IOException {

             String line = value.toString();

             for(Path cacheFile: dates){

                    output.collect(new Text(line), new Text(cacheFile.getName()));

                }



                }


            }

Answer 1

在您的configure()方法中尝试以下方法:

List<String []> lines; 
Path[] files = new Path[0];

public void configure(JobConf conf) {
    lines = new ArrayList<>();
    BufferedReader SW;
    try {
        files = DistributedCache.getLocalCacheFiles(conf);
        SW = new BufferedReader(new FileReader(files[0].toString()));
        String line;
        while ((line = SW.readLine()) != null) {
           lines.add(line.split(",")); //now, each lines entry is a String array, with each element being a column
        }
        SW.close();

    } catch (IOException ioe) {
        System.err.println("Caught exception while getting cached files: " +   
        StringUtils.stringifyException(ioe));
    }
}

这样，您将在变量 lines中的分布式缓存中拥有文件的内容(在本例中为第一个文件)。每个 lines条目均表示一个String数组，该数组用'，'分隔。因此，第一行的第一列是 lines.get(0)[0]，第二行的第三行是 lines.get(1)[2]，依此类推。