java 8流图检查以前的元素

Question

我有一个问题，我有一个正在阅读的大文本文件，我想要其中的单词列表并在其中找到特定的词对。

我的数据集的一个例子是:

A random text file . I am <pair-starter> first second <pair-ender> and it goes on and on,
and hopefully it ends .

现在我用类似流的方式读取文件

List<String> words = Files.lines(Paths.get(filename), Charset.forName("UTF-8")).
                     .map(line -> line.split("[\\s]+"))
                     .flatMap(Arrays::stream)
                     .filter(this::filterPunctuation) //This removes the dot in example
                     .map(this::removePunctuation) //This removes the comma
                     //I think the method should be added here
                     .filter(this::removePairSpesifics) //To remove pair starter and ender
                     .collect(Collectors.toList());

现在有了这段代码，我可以得到干净的单词，我得到一个包含 "A"、"random"、"text"、"file"、"I"、"am"、"first"的列表， “second”、“and”、“it”、“goes”、“on”、“and”、“on”、“and”、“hopefully”、“it”、“ends”但我也想得到一个包含对的 HashMap ，我想知道是否可以在上面的流中添加一个新方法。无法从谷歌找到任何接近我想要的东西，提前致谢。

接近我脑海中的方法是

private boolean pairStarted = false;
private String addToHashMap(String element){
    if previous element was pair starter
        pairStarted = true;
    else if pairStarted and element is not pairEnder
        MyPreviouslyConstructedHashMap.put(the previous one, element);
    else if element is pairEnder
        pairStarted = false;
    return element; 
} //This function will not change anything from the list as it returns the elements
  //But it'll add the hashmap first-second pair

---

我目前的解决方案是:

List<String> words = Files.lines(Paths.get(filename), Charset.forName("UTF-8")).
                     .map(line -> line.split("[\\s]+"))
                     .flatMap(Arrays::stream)
                     .filter(this::filterPunctuation)
                     .map(this::removePunctuation)
                     .collect(Collectors.toList()); //Now not using removePairSpesifics 
//as I need to check for them.
for(int i=words.size()-1; i>=0; i--) {
    if(words.get(i).equals("<pair-ender>")){ //checking from end to modify in the loop
        pairs.put(words.get(i-2), words.get(i-1));
        i = i-4;
        words.remove(i+1);
        words.remove(i+4);
    }
}

我想学习的是了解它是否可以在我将值读入列表的同一流中解决。

Answer 1

起初，我尝试将拆分分成两个拆分，结果非常好:

public void split(Stream<String> lines)
{
    Pattern pairFinder = Pattern.compile("<pair-starter|pair-ender>");
    Pattern spaceFinder = Pattern.compile("[\\s]+");

    Map<String, String> pairs = new HashMap<>();

    List<String> words = lines.flatMap(pairFinder::splitAsStream).flatMap(pairOrNoPair -> {
        if (pairOrNoPair.startsWith(">") && pairOrNoPair.endsWith("<"))
        {
            pairOrNoPair = pairOrNoPair.replaceAll("> +| +<", "");

            String[] pair = spaceFinder.split(pairOrNoPair);
            pairs.put(pair[0], pair[1]);
            return Arrays.stream(pair);
        }
        else
        {
            return spaceFinder.splitAsStream(pairOrNoPair.trim());
        }
    })
                              .filter(this::filterPunctuation) // This removes the dot in example
                              .map(this::removePunctuation) // This removes the comma
                              .collect(Collectors.toList());

    System.out.println(words);
    System.out.println(pairs);
}

// Output
// [A, random, text, file, I, am, first, second, and, it, goes, on, and, on, and, hopefully, it, ends]
// {first=second}

boolean filterPunctuation(String s)
{
    return !s.matches("[,.?!]");
}

String removePunctuation(String s)
{
    return s.replaceAll("[,.?!]", "");
}

这里发生了什么？首先，我们将线分为成对和非成对。对于其中的每一个，我们检查它们是否是一对。如果是这样，我们删除标记并将该对添加到列表中。在任何情况下，我们都会按空格拆分块，将其展平，然后逐字处理。

但是这个实现只是逐行处理输入。

---

要解决多线对的问题，我们可以尝试自定义 Collector 方法。看看这个相当快速和肮脏的尝试:

String t1 = "I am <pair-starter> first second <pair-ender>, <pair-starter> and";
String t2 = " hopefully <pair-ender> it ends .";
split(Stream.of(t1, t2));

public void split(Stream<String> lines)
{
    PairResult result = lines.flatMap(Pattern.compile("[\\s]+")::splitAsStream)
                             .map(word -> word.replaceAll("[,.?!]", ""))
                             .filter(word -> !word.isEmpty())
                             .collect(new PairCollector());

    System.out.println(result.words);
    System.out.println(result.pairs);
}

// Output
// [I, am, first, second, and, hopefully, it, ends]
// {and=hopefully, first=second}

class PairCollector
    implements Collector<String, PairResult, PairResult>
{
    @Override
    public Supplier<PairResult> supplier()
    {
        return PairResult::new;
    }

    @Override
    public BiConsumer<PairResult, String> accumulator()
    {
        return (result, word) -> {
            if ("<pair-starter>".equals(word))
            {
                result.inPair = true;
            }
            else if ("<pair-ender>".equals(word))
            {
                if (result.inPair)
                {
                    result.pairs.put(result.words.get(result.words.size() - 2),
                                     result.words.get(result.words.size() - 1));
                    result.inPair = false;
                }
                else
                {
                    // starter must be in another result, keep ender for combiner
                    result.words.add(word);
                }
            }
            else
            {
                result.words.add(word);
            }
        };
    }

    @Override
    public BinaryOperator<PairResult> combiner()
    {
        return (result1, result2) -> {
            // add completed pairs
            result1.pairs.putAll(result2.pairs);

            // use accumulator to finish split pairs
            BiConsumer<PairResult, String> acc = accumulator();
            result2.words.forEach(word2 -> acc.accept(result1, word2));

            return result1;
        };
    }

    @Override
    public Function<PairResult, PairResult> finisher()
    {
        return Function.identity();
    }

    @Override
    public Set<Characteristics> characteristics()
    {
        return new HashSet<>(Arrays.asList(Characteristics.IDENTITY_FINISH));
    }
}

class PairResult
{
    public boolean                   inPair;
    public final List<String>        words = new ArrayList<>();
    public final Map<String, String> pairs = new HashMap<>();
}

这个收集器逐字接受，并存储一些内部状态来跟踪对。它甚至应该适用于并行流，将单独的单词流组合成一个结果。