java - 如何防止分块数据处理的while循环中的代码重复？

Question

我正在读取一个文件，收集处理过的行，并在每个收集 block 之后将它们分批写入(例如写入文件或数据库)。

当循环终止(= 文件被完全读取)时，我必须再次调用 writer。否则我不会 catch 最后一 block 。

问题:我能否以某种方式改进代码以防止重复额外的 write() 调用？

List<String> collect = new ArrayList<>();

String line;
while ((line = reader.read()) != null) {
    String processed = processline(line);
    collect.add(processed);

    //write each x chunks to file
    if (collect.size() % 1000 == 0) {
        writer.write(collect);
        collect = new ArrayList<>();
    }
}

//can I prevent repetition here?
if (!collect.isEmpty()) {
    writer.write(collect);
}

Answer 1

将缓冲逻辑(因为这就是您正在做的事情，缓冲)封装在一个单独的类中。但是当缓冲区太大时，您总是必须写入，并且当您完成读取时。

class BufferingWriter implements Closeable {
    private List<String> buffer = new ArrayList<>(1000);
    private MyWriter writer;

    public void write(String line) {
        buffer.add(line);
        if (buffer.size() >= 1000) {
            flush();
        }
    }

    public void flush() {
        writer.write(buffer);
        buffer.clear();
    }

    @Override
    public void close() throws IOException {
        flush();
        // TBD: Pass the close call onto MyWriter if that is possible
        // or otherwise flag this writer as closed
    }
}

List<String> collect = new ArrayList<>();
try (BufferingWriter bwriter = new BufferingWriter(writer)) {
    String line;
    while ((line = reader.read()) != null) {
        String processed = processline(line);
        bwriter.write(line);
    }
}