java - 如何删除非单词边界的停用词？

Question

我正在尝试使用我的停用词列表中的内容删除 txt 文件中的停用词。有些停用词已被删除，有些则没有。

例如这句话:“味道很好，不是吗？”应该有一个类似“味道不错”的输出，但我的代码输出:“味道不错不是吗”

我的停用词列表来自:https://www.ranks.nl/stopwords (长停用词列表)。

这是我的代码:

    public static void main(String[] args) {

    ArrayList sw = new ArrayList<>();

    try{
        FileInputStream fis = new FileInputStream("/Users/Dan/Desktop/DATA/stopwords.txt");

        byte b[] = new byte[fis.available()];
        fis.read(b);
            fis.close();

            String data[] = new String(b).split("\n");

        for(int i = 0; i < data.length; i++)
        {
            sw.add(data[i].trim());
        }
         FileInputStream fis2 = new FileInputStream("/Users/Dan/Desktop/DATA/cleandata.txt");

        byte bb[] = new byte[fis2.available()];
        fis2.read(bb);
            fis2.close();

            String data2[] = new String(bb).split("\n");



            for(int i = 0; i < data2.length; i++)

        {
            String file = "";
            String s[] = data2[i].split("\\s");
            for(int j = 0; j < s.length; j++)
            {
                if(!(sw.contains(s[j].trim().toLowerCase())))
                {
                    file=file + s[j] + " ";
                }

            }
            file = file.replaceAll("[^a-zA-Z\\s+]", "");

            System.out.println(file.replaceAll("\\s+", " ").toLowerCase() + "\n");

        }

    } catch(Exception a){
        a.printStackTrace();
    }

}

我该怎么办？我认为打印时遇到问题

file = file.replaceAll("[^a-zA-Z\\s+]", "");

System.out.println(file.replaceAll("\\s+", " ").toLowerCase() + "\n");

Answer 1

使用了两种不同的引号字符。停用词文件包含 doesn't 并且您的输入包含 doesn't。

由于引号不同，所以单词不匹配。

编辑:这是一个稍微重构的解决方案，它会生成正确的输出(如果您不在输入中使用奇怪的引号)。

import java.util.Arrays;
import java.util.HashSet;
import java.util.Scanner;
import java.util.Set;
import java.util.stream.Collectors;

public class StopWordsApp {

    // the platform-specific end of line token
    private static final String EOL = String.format("%n");

    private final Set<String> stopWords = new HashSet<>(Arrays.asList(readLines("stopwords.txt")));

    public static void main(String[] args) {
        StopWordsApp stopWordsApp = new StopWordsApp();
        String[] lines = readLines("cleandata.txt");
        printLines(stopWordsApp.removeStopWords(lines));
    }

    private String[] removeStopWords(String[] inputLines) {
        return Arrays.stream(inputLines)
                // map the String array to a Line object
                .map(Line::new)
                // map the Line to a String without stop words
                .map(this::removeStopWords)
                // convert the stream to an array
                .toArray(String[]::new);
    }

    private String removeStopWords(Line line) {
        return line.words().stream()
                // map the word to its normalized version
                .map(Word::normalized)
                // remove stop words
                .filter(n -> !stopWords.contains(n))
                // join into a String separated by spaces
                .collect(Collectors.joining(" "));
    }

    private static String[] readLines(String fileName) {
        return readFile(fileName).split(EOL);
    }

    private static String readFile(String fileName) {
        return new Scanner(StopWordsApp.class.getResourceAsStream(fileName), "UTF-8").useDelimiter("\\A").next();
    }

    private static void printLines(String[] lines) {
        for (String line : lines) {
            System.out.println(line);
        }
    }
}

我为一条线提取了单独的类:

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class Line {

    private final List<Word> words;

    public Line(String input) {
        String[] wordInputs = input.split("\\s+");
        words = Arrays.stream(wordInputs)
                // remove empty Strings
                .filter(v -> !v.isEmpty())
                // map String to a Word object
                .map(Word::new)
                // collect into a List
                .collect(Collectors.toList());
    }

    public List<Word> words() {
        return words;
    }

}

..一句话:

public class Word {

    private final String normalized;

    public Word(String input) {
        normalized = input
                // convert to lower case
                .toLowerCase()
                // remove everything that's not a lower case letter or a quote
                // (the stopwords file only contains lower case letters and quotes)
                .replaceAll("[^a-z']", "")
                // replace consecutive white space with a single space
                .replaceAll("\\s+", " ")
                // trim any white space at the edges
                .trim();
    }

    public String normalized() {
        return normalized;
    }

}

...以及自定义(运行时)异常:

public class StopWordsException extends RuntimeException {
    public StopWordsException(Exception e) {
        super(e);
    }
}

我到处都使用了 Java 8 流，并添加了注释来解释发生了什么。

输入:

味道不错，不是吗？

输出为:

味道不错

附注文件“stopwords.txt”和“cleandata.txt”需要与 StopWordsApp 类位于同一包中。