python - Lexer 对 SKIP 和 NEWLINE 值进行标记。如何隐藏它们？

Question

下面的词法分析器代码会在应该隐藏它们的地方生成“SKIP”和“NEWLINE”值。有人可以告诉我应该做什么来隐藏它们吗？

        tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
        get_token = re.compile(tok_regex, re.DOTALL).match
        mo = get_token(self.input, self.p)

        self.p = mo.end()

        if self.p >= len(self.input):
            self.c = "EOF"
        else:
            if mo is not None:
                self.c = mo.lastgroup
                self.val = mo.group(self.c)
                if self.c == 'NEWLINE':
                    self.line += 1

    def next_token(self):

        keywords = {'digraph', 'subgraph', 'node', 'edge', 'strict', 'graph'}
        self.tokenize()
        if self.c != 'SKIP' and self.c != "NEWLINE":

            if self.c == 'ID' and self.val in keywords:
                self.c = self.val
        return TokenTypes.Token(self.c, self.val)
        # if self.p != len(self.input):
        #    raise RuntimeError('Unexpected character %r on line %d' %(self.input[self.p], self.line))

def main():
    statements = '''
        digraph G {
            main [shape=box]; 
        }
    '''
    lexer = Lexer(statements)
    t = lexer.next_token()
    i = 0
    while (t.typ != "EOF"):
        i += 1
        print(t)
        t = lexer.next_token()

if __name__ == "__main__":
    sys.exit(main())

我试图通过从上面的词法分析器导入标记来生成点语法的抽象语法树。但是不需要的 SKIP 和 NEWLINE 标记造成了麻烦。

这是一个示例输入:

digraph G { main [shape=box]; }

产生输出:

NEWLINE digraph SKIP ID SKIP LBRACE NEWLINE ID SKIP LBRACKET ID EQUAL ID RBRACKET END SKIP NEWLINE SKIP RBRACE

但它应该输出:

digraph ID LBRACE ID LBRACKET ID EQUAL ID RBRACKET END RBRACE

Answer 1

我相信您希望在遇到 NEWLINE 或 SKIP 时更改 next_token() 方法来解析另一个标记。示例:

def next_token(self):
    keywords = {'digraph', 'subgraph', 'node', 'edge', 'strict', 'graph'}

    # Parse next token.
    self.tokenize()
    while self.c == "SKIP" or self.c == "NEWLINE":
        # Discard this token and parse another token.
        self.tokenize()

    if self.c == 'ID' and self.val in keywords:
        self.c = self.val

    return TokenTypes.Token(self.c, self.val)
    # if self.p != len(self.input):
    #    raise RuntimeError('Unexpected character %r on line %d' %(self.input[self.p], self.line))