python - 在 python 中究竟如何生成 DEDENT token ？

Question

我正在阅读关于 lexical analysis of python 的文档，其中描述了如何生成 INDENT 和 DEDENT token 的类(class)。我在此处发布了描述。

The indentation levels of consecutive lines are used to generate INDENT and DEDENT tokens, using a stack, as follows.

Before the first line of the file is read, a single zero is pushed on the stack; this will never be popped off again. The numbers pushed on the stack will always be strictly increasing from bottom to top. At the beginning of each logical line, the line’s indentation level is compared to the top of the stack. If it is equal, nothing happens. If it is larger, it is pushed on the stack, and one INDENT token is generated. If it is smaller, it must be one of the numbers occurring on the stack; all numbers on the stack that are larger are popped off, and for each number popped off a DEDENT token is generated. At the end of the file, a DEDENT token is generated for each number remaining on the stack that is larger than zero.

我试图理解 DEDENT 部分，但未能理解，有人能给出比所引用的更好的解释吗？

Answer 1

由于 Python 有时比英语更容易，这里是此描述的 Python 粗略翻译。你可以看到真实世界的解析器(我自己写的)是这样工作的 here .

import re
code = """
for i in range(10):
   if i % 2 == 0:
     print(i)
   print("Next number")
print("That's all")

for i in range(10):
   if i % 2 == 0:
       print(i)
print("That's all again)

for i in range(10):
   if i % 2 == 0:
      print(i)
  print("That's all")
"""
def get_indent(s) -> int:
    m = re.match(r' *', s)
    return len(m.group(0))
def add_token(token):
    print(token)
INDENT="indent"
DEDENT="dedent"
indent_stack = [0]
# Before the first line of the file is read, a single zero is pushed on the stack
for line in code.splitlines():
    print("processing line:", line)
    indent = get_indent(line)
    # At the beginning of each logical line, the line’s 
    # indentation level is compared to the top of the stack. 
    if indent > indent_stack[-1]:
        # If it is larger, it is pushed on the stack, 
        # and one INDENT token is generated.
        add_token(INDENT)
        indent_stack.append(indent)
    elif indent < indent_stack[-1]:
        while indent < indent_stack[-1]:
            #  If it is smaller, ...
            # all numbers on the stack that are larger are popped off,
            # and for each number popped off a DEDENT token is generated.
            add_token(DEDENT)
            indent_stack.pop()
        if indent != indent_stack[-1]:
            # it must be one of the numbers occurring on the stack; 
            raise IndentationError
while indent_stack[-1]>0:
     # At the end of the file, a DEDENT token is generated for each number 
     # remaining on the stack that is larger than zero.
     add_token(DEDENT)
     indent_stack.pop()

这是输出:

processing line: 
processing line: for i in range(10):
processing line:    if i % 2 == 0:
indent
processing line:      print(i)
indent
processing line:    print("Next number")
dedent
processing line: print("That's all")
dedent
processing line: 
processing line: for i in range(10):
processing line:    if i % 2 == 0:
indent
processing line:        print(i)
indent
processing line: print("That's all again)
dedent
dedent
processing line: 
processing line: for i in range(10):
processing line:    if i % 2 == 0:
indent
processing line:       print(i)
indent
processing line:   print("That's all")
dedent
dedent
  File "<string>", line unknown
IndentationError