hadoop - 分布式缓存Hadoop无法检索文件内容

Question

我正在获取一些类似于值的垃圾，而不是要用作分布式缓存的文件中的数据。

作业配置如下:

Configuration config5 = new Configuration();
JobConf conf5 = new JobConf(config5, Job5.class);
conf5.setJobName("Job5");
conf5.setOutputKeyClass(Text.class);
conf5.setOutputValueClass(Text.class);
conf5.setMapperClass(MapThree4c.class);
conf5.setReducerClass(ReduceThree5.class);
conf5.setInputFormat(TextInputFormat.class);
conf5.setOutputFormat(TextOutputFormat.class);


DistributedCache.addCacheFile(new URI("/home/users/mlakshm/ap1228"), conf5);
FileInputFormat.setInputPaths(conf5, new Path(other_args.get(5)));
FileOutputFormat.setOutputPath(conf5, new Path(other_args.get(6)));

JobClient.runJob(conf5);

在Mapper中，我有以下代码:

public class MapThree4c extends MapReduceBase implements Mapper<LongWritable, Text, 
Text, Text >{
private Set<String> prefixCandidates = new HashSet<String>();

Text a = new Text();
public void configure(JobConf conf5) {

Path[] dates = new Path[0];
try {
        dates = DistributedCache.getLocalCacheFiles(conf5);
        System.out.println("candidates: "+candidates);
        String astr = dates.toString();
        a = new Text(astr);

      } catch (IOException ioe) {
        System.err.println("Caught exception while getting cached files: " +   
      StringUtils.stringifyException(ioe));
      }


  }




   public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, 
   Reporter reporter) throws IOException {

     String line = value.toString();
     StringTokenizer st = new StringTokenizer(line);
     st.nextToken();
     String t = st.nextToken();
     String uidi = st.nextToken();
     String uidj = st.nextToken();

     String check = null;

     output.collect(new Text(line), a);



        }


    }

我从此映射器获得的输出值为:[Lorg.apache.hadoop.fs.Path; @ 786c1a82
而不是分布式缓存文件中的值。

Answer 1

看起来就像您在数组上调用toString()时所得到的，并且如果您查看DistributedCache.getLocalCacheFiles()的javadocs，它就是返回的内容。如果您需要实际读取缓存中文件的内容，则可以使用标准的Java API打开/读取它们。