python - ply 中的特殊情况词法分析器规则

Question

有没有办法对层词法分析器规则进行特殊处理？

t_IDENT     = r'[a-zA-Z_][0-9a-zA-Z_]*'
t_OPERATOR  = r'[<>=/*+-]+'
t_DEFINE    = r'='
t_PRODUCES  = r'=>'

我想将运算符定义为列出的字符的任意组合，异常(exception) = 和 => 有其自己的特殊情况。例如:

a + b
# IDENT('a') OPERATOR('+') IDENT('b') 

a ++=--> b
# IDENT('a') OPERATOR('++=-->') IDENT('b') 

a == b
# IDENT('a') OPERATOR('==-->') IDENT('b') 

a => b
# IDENT('a') PRODUCES('=>') IDENT('b') 

a = b
# IDENT('a') DEFINE('=') IDENT('b') 

a >= b
# IDENT('a') OPERATOR('>=') IDENT('b') 

a <=> b
# IDENT('a') OPERATOR('<=>') IDENT('b') 

Answer 1

是的，您获得 OPERATOR 标记而不是预期的 PRODUCES/DEFINE 的原因是 PLY 词法分析器的标记优先规则:

Internally, lex.py uses the re module to do its patten matching. When building the master regular expression, rules are added in the following order:

1. All tokens defined by functions are added in the same order as they appear in the lexer file.
2. Tokens defined by strings are added next by sorting them in order of decreasing regular expression length (longer expressions are added first).

只需将某些规则转换为函数即可:

def t_DEFINE(t):
    r'='
    return t

def t_PRODUCES(t):
    r'=>'
    return t