java - 如何解决 "java.lang.OutOfMemoryError: Java heap space"问题？

Question

我正在编写一些代码来将一个非常大的平面文本文件解析为持久保存到数据库的对象。这适用于文件的各个部分(即，如果我“顶”前 2000 行)，但是当我尝试处理该文件时，我遇到了 java.lang.OutOfMemoryError: Java heap space 错误完整文件。

我正在使用 BufferedReader 逐行读取文件，我的印象是这不需要将整个文本文件加载到内存中。希望我的代码是不言自明的。我已经通过 Eclipse Memory Analyser 运行了我的代码，它告诉我:

The thread java.lang.Thread @ 0x27ee0478 main keeps local variables with total size 69,668,888 (98.76%) bytes.
The memory is accumulated in one instance of "char[]" loaded by "<system class loader>"**

非常感谢有用的评论!

乔纳森

public ArrayList<Statement> parseGMIFile(String filePath)
            throws IOException {

        ArrayList<Statement> statements = new ArrayList<Statement>();

        // Statement Properties
        String sAccount = "";
        String sOffice = "";
        String sFirm = "";
        String sDate1 = "";
        String sDate2 = "";
        Date date = new Date();
        StringBuffer sData = new StringBuffer();
        BufferedReader in = new BufferedReader(new FileReader(filePath));
        String line;
        String prevCode = "";
        int lineCounter = 1;
        int globalLineCounter = 1;

        while ((line = in.readLine()) != null) {

                // We extract the GMI code from the end of the first line
                String newCode = line.substring(GMICODE_START_POS).trim();

                // Extract date
                if (newCode.equals(prevCode)) {

                    if (lineCounter == DATE_LINE) { 
                        sDate1 = line.substring(DATE_START_POS, DATE_END_POS).trim();}

                    if (lineCounter == DATE_LINE2) {
                        sDate2 = line.substring(DATE_START_POS, DATE_END_POS).trim();}

                    if (sDate1.equals("")){
                        sDate1 = sDate2;}
                        SimpleDateFormat formatter=new SimpleDateFormat("MMM dd, yyyy");
                        try {
                            date=formatter.parse(sDate1);

                        } catch (ParseException e) {

                            e.printStackTrace();
                        }                   



                    sFirm = line.substring(FIRM_START_POS, FIRM_END_POS);
                    sOffice = line.substring(OFFICE_START_POS, OFFICE_END_POS);
                    sAccount = line.substring(ACCOUNT_START_POS,
                            ACCOUNT_END_POS);
                    lineCounter++;
                    globalLineCounter++;
                    sData.append(line.substring(0, END_OF_DATA)).append("\n");

                } else {

                    // Instantiate New Statement Object
                    Statement stmt = new Statement(sAccount, sOffice, sFirm,
                            date, sData.toString());


                    // Add to collection
                    statements.add(stmt);

                    // log.info("-----------NEW STATEMENT--------------");
                    sData.setLength(0);
                    lineCounter = 1;
                }
                prevCode = newCode;
        }
        return statements;
    }

STACKTRACE: Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbPopulator' defined in class path resource [app-context.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: Java heap space    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1401)    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512)    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450)    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290)    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287)    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:189)    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:557)    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:842)    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:416)    at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)    at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)    at Main.main(Main.java:11)Caused by: java.lang.OutOfMemoryError: Java heap space    at java.util.Arrays.copyOf(Arrays.java:2882)    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)    at java.lang.StringBuffer.append(StringBuffer.java:224)    at services.GMILogParser.parseGMIFile(GMILogParser.java:133)    at services.DBPopulator.init(DBPopulator.java:27)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)    at java.lang.reflect.Method.invoke(Method.java:597)    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1529)    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1468)    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1398)    ... 12 more

Answer 1

在启动参数中添加更多内存是一个错误。这些参数具有广泛的应用范围。并且可能会因增加 gc 次数而受到惩罚。此外，您可能事先不知道尺寸。

您使用MemoryMappedFiles并查看java.nio.*这样做。这样做就可以边读边加载，内存就不放在普通内存空间了。

通过低层次的阅读，你可以将其分成不同长度的 block 。速度很重要。如果您的文件很大，则可能需要花费太多时间来读取它。而且，您在 JVM 中存储的对象数量会使 GC 正常工作，并且应用程序会变慢。来自java引用:

• 字节缓冲区可以分配为直接缓冲区，在这种情况下，Java 虚拟机将尽最大努力执行 native I/O 操作直接在它上面。

• 可以通过将文件的某个区域直接映射到内存来创建字节缓冲区，在这种情况下，可以使用 MappedByteBuffer 类中定义的一些附加文件相关操作。

• 字节缓冲区提供对其内容的访问，作为任何非 boolean 基元类型的异构或同构二进制数据序列，采用大端字节序或小端字节顺序.