python - NLTK 关系提取不返回任何内容

Question

我最近致力于使用 nltk 从文本中提取关系。所以我构建了一个示例文本:“Tom is the cofounder of Microsoft.”并使用以下程序进行测试并且不返回任何内容。我不明白为什么。

我使用的是 NLTK 版本:3.2.1，python 版本:3.5.2。

这是我的代码:

import re
import nltk
from nltk.sem.relextract import extract_rels, rtuple
from nltk.tokenize import sent_tokenize, word_tokenize


def test():
    with open('sample.txt', 'r') as f:
        sample = f.read()   # "Tom is the cofounder of Microsoft"

    sentences = sent_tokenize(sample)
    tokenized_sentences = [word_tokenize(sentence) for sentence in sentences]
    tagged_sentences = [nltk.tag.pos_tag(sentence) for sentence in tokenized_sentences]

    OF = re.compile(r'.*\bof\b.*')

    for i, sent in enumerate(tagged_sentences):
        sent = nltk.chunk.ne_chunk(sent) # ne_chunk method expects one tagged sentence
        rels = extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10) 
        for rel in rels:
            print('{0:<5}{1}'.format(i, rtuple(rel)))

if __name__ == '__main__':
    test()

---

1。经过一些调试，如果发现当我将输入更改为

"Gates was born in Seattle, Washington on October 28, 1955. "

nltk.chunk.ne_chunk() 的输出是:

(S (PERSON Gates/NNS) was/VBD born/VBN in/IN (GPE Seattle/NNP) ,/, (GPE Washington/NNP) on/IN October/NNP 28/CD ,/, 1955/CD ./.)

test() 返回:

[PER: 'Gates/NNS'] 'was/VBD born/VBN in/IN' [GPE: 'Seattle/NNP']

2。在我将输入更改为:

"Gates was born in Seattle on October 28, 1955. "

test() 不返回任何内容。

3。我深入研究了 nltk/sem/relextract.py 并发现这很奇怪

输出是由函数引起的:semi_rel2reldict(pairs, window=5, trace=False)，只有当len(pairs) > 2时才返回结果，这就是为什么当一个句子少于三个NE时会返回None。

这是错误还是我以错误的方式使用了 NLTK？

Answer 1

首先，用 ne_chunk 分块 NE，成语看起来像这样

>>> from nltk import ne_chunk, pos_tag, word_tokenize
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')])])

(另见 https://stackoverflow.com/a/31838373/610569)

接下来让我们看看extract_rels function .

def extract_rels(subjclass, objclass, doc, corpus='ace', pattern=None, window=10):
    """
    Filter the output of ``semi_rel2reldict`` according to specified NE classes and a filler pattern.
    The parameters ``subjclass`` and ``objclass`` can be used to restrict the
    Named Entities to particular types (any of 'LOCATION', 'ORGANIZATION',
    'PERSON', 'DURATION', 'DATE', 'CARDINAL', 'PERCENT', 'MONEY', 'MEASURE').
    """

当你调用这个函数时:

extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10)

它依次执行 4 个过程。

1。它检查您的 subjclass 和 objclass 是否有效

即https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L202 :

if subjclass and subjclass not in NE_CLASSES[corpus]:
    if _expand(subjclass) in NE_CLASSES[corpus]:
        subjclass = _expand(subjclass)
    else:
        raise ValueError("your value for the subject type has not been recognized: %s" % subjclass)
if objclass and objclass not in NE_CLASSES[corpus]:
    if _expand(objclass) in NE_CLASSES[corpus]:
        objclass = _expand(objclass)
    else:
        raise ValueError("your value for the object type has not been recognized: %s" % objclass)

2。它从您的 NE 标记输入中提取“对”:

if corpus == 'ace' or corpus == 'conll2002':
    pairs = tree2semi_rel(doc)
elif corpus == 'ieer':
    pairs = tree2semi_rel(doc.text) + tree2semi_rel(doc.headline)
else:
    raise ValueError("corpus type not recognized")

现在让我们看看给定你的输入语句 Tom is the cofounder of Microsoft，tree2semi_rel() 返回什么:

>>> from nltk.sem.relextract import tree2semi_rel, semi_rel2reldict
>>> from nltk import word_tokenize, pos_tag, ne_chunk
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]

因此它返回一个包含 2 个列表的列表，第一个内部列表由一个空白列表和包含“PERSON”标签的 Tree 组成。

[[], Tree('PERSON', [('Tom', 'NNP')])] 

第二个列表由短语 is the cofounder of 和包含“ORGANIZATION”的 Tree 组成。

让我们继续。

3。 extract_rel 然后尝试将这些对更改为某种关系字典

reldicts = semi_rel2reldict(pairs)

如果我们查看 semi_rel2reldict 函数返回的示例语句，我们会发现这是空列表返回的地方:

>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[]

那么让我们看看semi_rel2reldict的代码https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L144 :

def semi_rel2reldict(pairs, window=5, trace=False):
    """
    Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
    stores information about the subject and object NEs plus the filler between them.
    Additionally, a left and right context of length =< window are captured (within
    a given input sentence).
    :param pairs: a pair of list(str) and ``Tree``, as generated by
    :param window: a threshold for the number of items to include in the left and right context
    :type window: int
    :return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
    :rtype: list(defaultdict)
    """
    result = []
    while len(pairs) > 2:
        reldict = defaultdict(str)
        reldict['lcon'] = _join(pairs[0][0][-window:])
        reldict['subjclass'] = pairs[0][1].label()
        reldict['subjtext'] = _join(pairs[0][1].leaves())
        reldict['subjsym'] = list2sym(pairs[0][1].leaves())
        reldict['filler'] = _join(pairs[1][0])
        reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
        reldict['objclass'] = pairs[1][1].label()
        reldict['objtext'] = _join(pairs[1][1].leaves())
        reldict['objsym'] = list2sym(pairs[1][1].leaves())
        reldict['rcon'] = _join(pairs[2][0][:window])
        if trace:
            print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
        result.append(reldict)
        pairs = pairs[1:]
    return result

semi_rel2reldict() 做的第一件事是检查 tree2semi_rel() 的输出中有超过 2 个元素的地方，而您的例句没有:

>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> len(tree2semi_rel(chunked))
2
>>> len(tree2semi_rel(chunked)) > 2
False

啊哈，这就是 extract_rel 没有返回任何内容的原因。

现在的问题是如何让 extract_rel() 返回一些东西，即使是来自 tree2semi_rel() 的 2 个元素？ 这甚至可能吗？

让我们试试不同的句子:

>>> text = "Tom is the cofounder of Microsoft and now he is the founder of Marcohard"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')]), ('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN'), Tree('PERSON', [('Marcohard', 'NNP')])])
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])], [[('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN')], Tree('PERSON', [('Marcohard', 'NNP')])]]
>>> len(tree2semi_rel(chunked)) > 2
True
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': 'and/CC now/RB he/PRP is/VBZ the/DT', 'subjtext': 'Tom/NNP'})]

但这只能证实当 tree2semi_rel 返回成对的 <2 时 extract_rel 无法提取。如果我们删除 while 的条件会发生什么len(对) > 2?

为什么我们不能while len(pairs) > 1？

如果我们仔细观察代码，我们会看到填充 reldict 的最后一行 https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L169 :

reldict['rcon'] = _join(pairs[2][0][:window])

它尝试访问 pairs 的第三个元素，如果 pairs 的长度为 2，您将得到一个 IndexError .

那么，如果我们删除 rcon 键并将其简单地更改为 while len(pairs) >= 2 会发生什么？

为此，我们必须覆盖 semi_rel2redict() 函数:

>>> from nltk.sem.relextract import _join, list2sym
>>> from collections import defaultdict
>>> def semi_rel2reldict(pairs, window=5, trace=False):
...     """
...     Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
...     stores information about the subject and object NEs plus the filler between them.
...     Additionally, a left and right context of length =< window are captured (within
...     a given input sentence).
...     :param pairs: a pair of list(str) and ``Tree``, as generated by
...     :param window: a threshold for the number of items to include in the left and right context
...     :type window: int
...     :return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
...     :rtype: list(defaultdict)
...     """
...     result = []
...     while len(pairs) >= 2:
...         reldict = defaultdict(str)
...         reldict['lcon'] = _join(pairs[0][0][-window:])
...         reldict['subjclass'] = pairs[0][1].label()
...         reldict['subjtext'] = _join(pairs[0][1].leaves())
...         reldict['subjsym'] = list2sym(pairs[0][1].leaves())
...         reldict['filler'] = _join(pairs[1][0])
...         reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
...         reldict['objclass'] = pairs[1][1].label()
...         reldict['objtext'] = _join(pairs[1][1].leaves())
...         reldict['objsym'] = list2sym(pairs[1][1].leaves())
...         reldict['rcon'] = []
...         if trace:
...             print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
...         result.append(reldict)
...         pairs = pairs[1:]
...     return result
... 
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]

啊!它有效，但 extract_rels() 中还有第 4 步。

4。它根据您提供给 pattern 参数的正则表达式执行 reldict 过滤器，https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L222 :

relfilter = lambda x: (x['subjclass'] == subjclass and
                       len(x['filler'].split()) <= window and
                       pattern.match(x['filler']) and
                       x['objclass'] == objclass)

现在让我们用破解版的 semi_rel2reldict 试试看:

>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]
>>> 
>>> pattern = re.compile(r'.*\bof\b.*')
>>> reldicts = semi_rel2reldict(tree2semi_rel(chunked))
>>> relfilter = lambda x: (x['subjclass'] == subjclass and
...                            len(x['filler'].split()) <= window and
...                            pattern.match(x['filler']) and
...                            x['objclass'] == objclass)
>>> relfilter
<function <lambda> at 0x112e591b8>
>>> subjclass = 'PERSON'
>>> objclass = 'ORGANIZATION'
>>> window = 5
>>> list(filter(relfilter, reldicts))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]

有效!现在让我们以元组形式查看它:

>>> from nltk.sem.relextract import rtuple
>>> rels = list(filter(relfilter, reldicts))
>>> for rel in rels:
...     print rtuple(rel)
... 
[PER: 'Tom/NNP'] 'is/VBZ the/DT cofounder/NN of/IN' [ORG: 'Microsoft/NNP']