python - 获得一种语法来阅读文本中的多个关键字

Question

我仍然认为自己是 pyparsing 的新手。我拼凑了 2 个快速语法，但都没有成功地完成我想做的事情。我正在尝试提出一种看起来非常简单的语法，但事实证明(至少对我而言)并不是那么微不足道。该语言有一个基本定义。它按关键字和正文分割。正文可以跨越多行。关键字位于前 20 个字符左右的行首，但以“;”结尾(没有引号)。所以我拼凑了一个快速演示程序，这样我就可以用几个语法进行测试。但是，当我尝试使用它们时，它们总是获得第一个关键字，但之后没有。

我附上了源代码作为示例和正在发生的输出。尽管这只是测试代码，但出于习惯我做了文档。在下面的示例中，两个关键字是 NOW；最后;理想情况下，我不希望关键字中包含分号。

我应该怎么做才能使这项工作有任何想法？

from pyparsing import *

def testString(text,grammar):
    """
    @summary: perform a test of a grammar
    2type text: text
    @param text: text buffer for input (a message to be parsed)
    @type grammar: MatchFirst or equivalent pyparsing construct
    @param grammar: some grammar defined somewhere else
    @type pgm: text
    @param pgm: typically name of the program, which invoked this function.
    @status: 20130802 CODED
    """
    print 'Input Text is %s' % text
    print 'Grammar is %s' % grammar
    tokens = grammar.parseString(text)
    print 'After parse string: %s' % tokens
    tokens.dump()
    tokens.keys()

    return tokens


def getText(msgIndex):
    """
    @summary: make a text string suitable for parsing
    @returns: returns a text buffer
    @type msgIndex: int
    @param msgIndex: a number corresponding to a text buffer to retrieve
    @status: 20130802 CODED
    """

    msg = [  """NOW; is the time for a few good ones to come to the aid
of new things to come for it is almost time for
a tornado to strike upon a small hill
when least expected.
lastly; another day progresses and
then we find that which we seek
and finally we will
find our happiness perhaps its closer than 1 or 2 years or not so
    """,
         '',
      ]

    return msg[msgIndex]

def getGrammar(grammarIndex):
    """
    @summary: make a grammar given an index
    @type: grammarIndex: int
    @param grammarIndex: a number corresponding to the grammar to be retrieved
    @Note: a good run will return 2 keys: NOW: and lastly:  and each key will have an associated body. The body is all
    words and text up to the next keyword or eof which ever is first.
    """
    kw = Combine(Word(alphas + nums) + Literal(';'))('KEY')
    kw.setDebug(True)
    body1 = delimitedList(OneOrMore(Word(alphas + nums)) +~kw)('Body')
    body1.setDebug(True)
    g1 = OneOrMore(Group(kw + body1))

    # ok start defining a new grammar (borrow kw from grammar).

    body2 = SkipTo(~kw, include=False)('BODY')
    body2.setDebug(True)

    g2 = OneOrMore(Group(kw+body2))
    grammar = [g1,
           g2,
          ]
    return grammar[grammarIndex]


if __name__ == '__main__':
    # list indices [ text, grammar ]
    tests = {1: [0,0],
         2: [0,1],
        }
    check = tests.keys()
    check.sort()
    for testno in check:
    print 'STARTING Test %d' % testno
    text = getText(tests[testno][0])
    grammar = getGrammar(tests[testno][1])
    tokens = testString(text, grammar)
    print 'Tokens found %s' % tokens
    print 'ENDING Test %d' % testno

输出如下所示:(使用 python 2.7 和 pyparsing 2.0.1)

    STARTING Test 1
    Input Text is NOW; is the time for a few good ones to come to the aid
    of new things to come for it is almost time for
    a tornado to strike upon a small hill
    when least expected.
    lastly; another day progresses and
    then we find that which we seek
    and finally we will
    find our happiness perhaps its closer than 1 or 2 years or not so

    Grammar is {Group:({Combine:({W:(abcd...) ";"}) {{W:(abcd...)}... ~{Combine:({W:(abcd...) ";"})}} [, {{W:(abcd...)}... ~{Combine:({W:(abcd...) ";"})}}]...})}...
    Match Combine:({W:(abcd...) ";"}) at loc 0(1,1)
    Matched Combine:({W:(abcd...) ";"}) -> ['NOW;']
    Match {{W:(abcd...)}... ~{Combine:({W:(abcd...) ";"})}} [, {{W:(abcd...)}... ~{Combine:({W:(abcd...) ";"})}}]... at loc 4(1,5)
    Match Combine:({W:(abcd...) ";"}) at loc 161(4,20)
    Exception raised:Expected W:(abcd...) (at char 161), (line:4, col:20)
    Matched {{W:(abcd...)}... ~{Combine:({W:(abcd...) ";"})}} [, {{W:(abcd...)}... ~{Combine:({W:(abcd...) ";"})}}]... -> ['is', 'the', 'time', 'for', 'a', 'few', 'good', 'ones', 'to', 'come', 'to', 'the', 'aid', 'of', 'new', 'things', 'to', 'come', 'for', 'it', 'is', 'almost', 'time', 'for', 'a', 'tornado', 'to', 'strike', 'upon', 'a', 'small', 'hill', 'when', 'least', 'expected']
    Match Combine:({W:(abcd...) ";"}) at loc 161(4,20)
    Exception raised:Expected W:(abcd...) (at char 161), (line:4, col:20)
    After parse string: [['NOW;', 'is', 'the', 'time', 'for', 'a', 'few', 'good', 'ones', 'to', 'come', 'to', 'the', 'aid', 'of', 'new', 'things', 'to', 'come', 'for', 'it', 'is', 'almost', 'time', 'for', 'a', 'tornado', 'to', 'strike', 'upon', 'a', 'small', 'hill', 'when', 'least', 'expected']]
    Tokens found [['NOW;', 'is', 'the', 'time', 'for', 'a', 'few', 'good', 'ones', 'to', 'come', 'to', 'the', 'aid', 'of', 'new', 'things', 'to', 'come', 'for', 'it', 'is', 'almost', 'time', 'for', 'a', 'tornado', 'to', 'strike', 'upon', 'a', 'small', 'hill', 'when', 'least', 'expected']]
    ENDING Test 1
    STARTING Test 2
    Input Text is NOW; is the time for a few good ones to come to the aid
    of new things to come for it is almost time for
    a tornado to strike upon a small hill
    when least expected.
    lastly; another day progresses and
    then we find that which we seek
    and finally we will
    find our happiness perhaps its closer than 1 or 2 years or not so

    Grammar is {Group:({Combine:({W:(abcd...) ";"}) SkipTo:(~{Combine:({W:(abcd...) ";"})})})}...
    Match Combine:({W:(abcd...) ";"}) at loc 0(1,1)
    Matched Combine:({W:(abcd...) ";"}) -> ['NOW;']
    Match SkipTo:(~{Combine:({W:(abcd...) ";"})}) at loc 4(1,5)
    Match Combine:({W:(abcd...) ";"}) at loc 4(1,5)
    Exception raised:Expected ";" (at char 7), (line:1, col:8)
    Matched SkipTo:(~{Combine:({W:(abcd...) ";"})}) -> ['']
    Match Combine:({W:(abcd...) ";"}) at loc 5(1,6)
    Exception raised:Expected ";" (at char 7), (line:1, col:8)
    After parse string: [['NOW;', '']]
    Tokens found [['NOW;', '']]
    ENDING Test 2

    Process finished with exit code 0

Answer 1

我对 TDD 很在行，但在这里，您的整个测试和备选方案选择基础架构确实阻碍了查看语法的位置以及正在发生的事情。如果我去掉所有额外的机制，我看到你的语法只是:

kw = Combine(Word(alphas + nums) + Literal(';'))('KEY')
body1 = delimitedList(OneOrMore(Word(alphas + nums)) +~kw)('Body')
g1 = OneOrMore(Group(kw + body1))

我看到的第一个问题是你对 body1 的定义:

body1 = delimitedList(OneOrMore(Word(alphas + nums)) +~kw)('Body')

你在正确的轨道上有一个负面的前瞻性，但要让它在 pyparsing 中工作，你必须把它放在表达式的开头，而不是最后。把它想象成“在我匹配另一个有效词之前，我会首先排除它是一个关键字。”:

body1 = delimitedList(OneOrMore(~kw + Word(alphas + nums)))('Body')

(顺便说一下，为什么这是一个 delimitedList？delimitedList 通常保留用于带有分隔符的真实列表，例如程序函数的逗号分隔参数。所有这确实是接受任何可能混入正文的逗号，应该使用标点符号列表更直接地处理。)

这是我的代码测试版本:

from pyparsing import *

kw = Combine(Word(alphas + nums) + Literal(';'))('KEY')
body1 = OneOrMore(~kw + Word(alphas + nums))('Body')
g1 = OneOrMore(Group(kw + body1))

msg = [  """NOW; is the time for a few good ones to come to the aid
of new things to come for it is almost time for
a tornado to strike upon a small hill
when least expected.
lastly; another day progresses and
then we find that which we seek
and finally we will
find our happiness perhaps its closer than 1 or 2 years or not so
    """,
             '',
          ][0]

result = g1.parseString(msg)
# we expect multiple groups, each containing "KEY" and "Body" names,
# so iterate over groups, and dump the contents of each
for res in result:
    print res.dump()

我仍然得到与您相同的结果，只是第一个关键字匹配。因此，为了查看断开连接发生的位置，我使用了 scanString，它不仅返回匹配的标记，还返回匹配标记的开始和结束:

result,start,end = next(g1.scanString(msg))
print len(msg),end

这给了我:

320 161

所以我看到我们在一个总长度为 320 的字符串中的位置 161 处结束，所以我将再添加一个打印语句:

print msg[end:end+10]

我得到:

.
lastly;

正文中的结尾句点是罪魁祸首。如果我从消息中删除它并再次尝试 parseString，我现在得到:

['NOW;', 'is', 'the', 'time', 'for', 'a', 'few', 'good', 'ones', 'to', 'come', 'to', 'the', 'aid', 'of', 'new', 'things', 'to', 'come', 'for', 'it', 'is', 'almost', 'time', 'for', 'a', 'tornado', 'to', 'strike', 'upon', 'a', 'small', 'hill', 'when', 'least', 'expected']
- Body: ['is', 'the', 'time', 'for', 'a', 'few', 'good', 'ones', 'to', 'come', 'to', 'the', 'aid', 'of', 'new', 'things', 'to', 'come', 'for', 'it', 'is', 'almost', 'time', 'for', 'a', 'tornado', 'to', 'strike', 'upon', 'a', 'small', 'hill', 'when', 'least', 'expected']
- KEY: NOW;
['lastly;', 'another', 'day', 'progresses', 'and', 'then', 'we', 'find', 'that', 'which', 'we', 'seek', 'and', 'finally', 'we', 'will', 'find', 'our', 'happiness', 'perhaps', 'its', 'closer', 'than', '1', 'or', '2', 'years', 'or', 'not', 'so']
- Body: ['another', 'day', 'progresses', 'and', 'then', 'we', 'find', 'that', 'which', 'we', 'seek', 'and', 'finally', 'we', 'will', 'find', 'our', 'happiness', 'perhaps', 'its', 'closer', 'than', '1', 'or', '2', 'years', 'or', 'not', 'so']
- KEY: lastly;

如果你想处理标点符号，我建议你添加如下内容:

PUNC = oneOf(". , ? ! : & $")

并将其添加到 body1:

body1 = OneOrMore(~kw + (Word(alphas + nums) | PUNC))('Body')