java - 获取一个单词在文本文件中出现的次数并将其链接到文本文件

Question

我目前有 3 个包含数据的文本文件

Textfile1
Hello World
Bye World

Textfile2
Hello World
Hello Second

如何得到结果

Hello {Textfile1 = 1, Textfile2 =2}
World {Textfile1 = 2, Textfile2 = 1}

目前我已经设法将我的 map 中的文字传递到我的 Reduce java 页面中。这就是我目前所处的位置。

public class Reduce extends Reducer<Text, Text, Text, Text> {
    HashMap<Text, Integer>input = new HashMap<Text, Integer>();

    public void reduce(Text key, Iterable<Text> values , Context context)
    throws IOException, InterruptedException {
        int sum = 0;
        for(Text val: values){
            String word = key.toString();
            Text filename;
            input.put(val,sum );
                if(//not sure what to write here){
                 
               }
            }
       context.write(new Text(key), input);
}

我的映射器代码

public class Map extends Mapper<LongWritable, Text, Text, Text> {

private Text file = new Text();
private Text word = new Text();
private String pattern= "^[a-z][a-z0-9]*$";//any lower case letter or number

public void map(LongWritable key, Text value,Context context) throws IOException, InterruptedException {
    
    InputSplit inputSplit = context.getInputSplit();
    String fileName = ((FileSplit)inputSplit).getPath().getName();
    file.set(fileName);
    String line = value.toString();
    StringTokenizer tokenizer = new StringTokenizer(line);
    while (tokenizer.hasMoreTokens()) {
       word.set(tokenizer.nextToken());
        
        String stringWord = word.toString().toLowerCase();
        if (stringWord.matches(pattern)){
            context.write(new Text(stringWord), new Text(fileName));
            
        }
    }
}

希望能得到一些帮助

Answer 1

在映射器的输出中，我们可以将文本文件名设置为键，将文件中的每一行设置为值。

可以使用 Mapper 类中的以下代码片段检索文件名。

FileSplit fileSplit = (FileSplit)context.getInputSplit();
String filename = fileSplit.getPath().getName();

然后在reducer中

public class Reduce extends Reducer<Text, Text, Text, Text> {
HashMap<Text, Integer>input = new HashMap<Text, Integer>();

public void reduce(Text key, Iterable<Text> values , Context context)
throws IOException, InterruptedException {
    int sum = 0;
    for(Text val: values){
        String word = val.toString(); -- processing each row
        String[] wordarray = word.split(' ');
        for(int i=0 ; i<wordarray.length; i++)
       {
        if(input.get(wordarray[i]) == null){
        input.put(wordarray[i],1);}
        else{
         int value =input.get(wordarray[i]) +1 ; 
         input.put(wordarray[i],value);
         }
       }     

   context.write(new Text(key), new Text(input.toString()));
}