java - 在 Java 中对大文件执行 "File to byte[]"时出现 OutOfMemoryError 错误？

Question

我尝试了各种代码将大型 CSV 文件(~300 MB)转换为 byte[]，但每次都失败，给出 Java 堆空间错误，如下所示:

184898 [jobLauncherTaskExecutor-1] DEBUG org.springframework.batch.core.step.tasklet.TaskletStep - Rollback for Error: java.lang.OutOfMemoryError: Java heap space 185000 [jobLauncherTaskExecutor-1] DEBUG org.springframework.transaction.support.TransactionTemplate - Initiating transaction rollback on application exception java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2367) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) at java.lang.StringBuffer.append(StringBuffer.java:237) at org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.format(PatternParser.java:419) at org.apache.log4j.PatternLayout.format(PatternLayout.java:506) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310) at org.apache.log4j.WriterAppender.append(WriterAppender.java:162) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) at org.apache.log4j.Category.callAppenders(Category.java:206) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:601) at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133) at org.apache.http.impl.conn.Wire.wire(Wire.java:77) at org.apache.http.impl.conn.Wire.output(Wire.java:107) at org.apache.http.impl.conn.LoggingSessionOutputBuffer.write(LoggingSessionOutputBuffer.java:76) at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:119) at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:115) at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98) at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108) at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122) at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271) at org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:227) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)

到目前为止，我已尝试使用以下版本的代码来将文件转换为 byte[]:

版本 1:核心 Java

    File file = new File(fileName);
    FileInputStream fin = null;
    byte fileContent[] = null;

    try {
        fin = new FileInputStream(file);

        fileContent = new byte[(int) file.length()];

        fin.read(fileContent);

    } catch (FileNotFoundException e) {
        System.out.println("File not found" + e);
    } catch (IOException ioe) {
        System.out.println("Exception while reading file " + ioe);
    } finally {
        try {
            if (fin != null) {
                fin.close();
            }
        } catch (IOException ioe) {
            System.out.println("Error while closing stream: " + ioe);
        }
    }

    return fileContent;

版本 2:Java 7 NIO

    Path path = Paths.get(fileName);

    byte[] data = null;

    try {
        data = Files.readAllBytes(path);
    } catch (IOException e) {
        e.printStackTrace();
    }

    return data;

版本 3:Apache Commons IO

    File file = new File(fileName);
    FileInputStream fis = null;
    byte fileContent[] = null;

    try {
        fis = new FileInputStream(file);

        fileContent = IOUtils.toByteArray(fis);

    } catch (FileNotFoundException e) {
        System.out.println("File not found" + e);
    } catch (IOException ioe) {
        System.out.println("Exception while reading file " + ioe);
    } finally {
        try {
            if (fis != null) {
                fis.close();
            }
        } catch (IOException ioe) {
            System.out.println("Error while closing stream: " + ioe);
        }
    }

    return fileContent;

版本 4:Google Guava

    File file = new File(fileName);
    FileInputStream fis = null;
    byte fileContent[] = null;

    try {
        fis = new FileInputStream(file);

        fileContent = ByteStreams.toByteArray(fis);

    } catch (FileNotFoundException e) {
        System.out.println("File not found" + e);
    } catch (IOException ioe) {
        System.out.println("Exception while reading file " + ioe);
    } finally {
        try {
            if (fis != null) {
                fis.close();
            }
        } catch (IOException ioe) {
            System.out.println("Error while closing stream: " + ioe);
        }
    }

    return fileContent;

版本 5:Apache.commons.io.FileUtils

File file = new File(fileName);

byte fileContent[] = null;

try {

    fileContent =  org.apache.commons.io.FileUtils.readFileToByteArray(file);

} catch (FileNotFoundException e) {
    System.out.println("File not found" + e);
} catch (IOException ioe) {
    System.out.println("Exception while reading file " + ioe);
}

return fileContent;

我什至将堆空间设置得相当大。我的外部 Tomcat 大约有 6 GB (5,617,772 K)，如任务管理器中的内存消耗所示。

对于前三个版本的代码，在命中此 byte[] 生成代码时，堆空间突然增加到 5 GB 以上，然后失败。对于 Google Guava，它看起来非常有前途，在命中 byte[] 生成代码后，内存消耗在相当长的一段时间内保持在 3.5 GB 左右，例如大约 10 分钟，然后突然跳到 5 GB 以上并失败。

我无法找到解决此问题的方法。有人可以帮我解决这个问题吗？任何对此的帮助将不胜感激。

Answer 1

加载到字节数组时，300MB 的文件不会消耗 6GB 的堆。仔细观察你的堆栈跟踪，似乎加载部分完全没问题。仅当您尝试使用 Log4j 记录某些内容时，才会抛出“java.lang.OutOfMemoryError:Java 堆空间”。

日志记录似乎源自第 3 方代码而不是您自己的代码，因此您可能无法更改正在记录的内容，但您绝对可以通过 Log4j 配置减少日志记录，尝试增加 org.apache.* 的日志级别(到 WARN、ERROR 或 FATAL)，然后就可以了。