java.lang.OutOfMemory错误: Java heap space Exception while Parsing of Large file using ANTLR

Question

我正在尝试在 ANTLR 版本 4 的帮助下解析日志文件。目前该文件由 10703 行组成，可能超过数百万行。 Java提示以下heap Exception。我将行数减少到 300 行。程序成功运行，但是当我将其行数设置为 400 时，它又开始出现堆异常。我也放大了java的堆内存，但没有成功。我也不知道这个问题是java的还是ANTLR的。

这是执行语法的程序:

public class parser {
    public static void main(String[] args) {
        System.out.println("Start");
        String fileName = "D:\\folder\\logs.out";
        File file = new File(fileName);
        FileInputStream fis = null;

        try {
        // Open the input file stream

        fis = new FileInputStream(file);

        // Create a CharStream that reads from standard input

        ANTLRInputStream input = new ANTLRInputStream(fis);

        GrammarOSBLexer lexer=new GrammarOSBLexer(input);

        TokenStream tokenStream=new CommonTokenStream(lexer);
        GrammarOSBParser parsr=new GrammarOSBParser(tokenStream);
        fis.close();

        try {
            parsr.logs();

         } catch (RecognitionException e) {

             e.printStackTrace();
         }

        System.out.println("done!");

    }catch (IOException e) {

        e.printStackTrace();
   }            

  } 

}

`

这里是异常(exception):

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.ArrayList.grow(Unknown Source)
    at java.util.ArrayList.ensureExplicitCapacity(Unknown Source)
    at java.util.ArrayList.ensureCapacityInternal(Unknown Source)
    at java.util.ArrayList.add(Unknown Source)
    at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:146)
    at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:137)
    at org.antlr.runtime.CommonTokenStream.consume(CommonTokenStream.java:68)
    at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:106)
    at com.javadude.antlr.tutorial.GrammarOSBParser.string(GrammarOSBParser.java:1269)
    at com.javadude.antlr.tutorial.GrammarOSBParser.random_messageText(GrammarOSBParser.java:350)
    at com.javadude.antlr.tutorial.GrammarOSBParser.messageTextTag(GrammarOSBParser.java:219)
    at com.javadude.antlr.tutorial.GrammarOSBParser.log(GrammarOSBParser.java:173)
    at com.javadude.antlr.tutorial.GrammarOSBParser.logs(GrammarOSBParser.java:111)
    at com.javadude.antlr.tutorial.parser.main(parser.java:91)
    ... 5 more

`

Answer 1

看起来将所有 token 缓冲在内存中效果不佳。因此，您应该寻找一种不进行缓冲的解决方案。 ANTLR4 带有 UnbufferedTokenStream更适合您的类(class)。另请参阅this question讨论如何使用这样的流以及它有哪些缺点(Sam Harwell + Terence Parr 做出了贡献)。