gpt4 book ai didi

python - 名词短语合并后如何获得句子的正确 pos 标签?

转载 作者:太空宇宙 更新时间:2023-11-04 04:09:28 24 4
gpt4 key购买 nike

我正在尝试合并句子中的名词短语 block ,然后获取合并文档中每个标记的 pos 标签。然而,对于每个合并的跨度,我似乎得到了跨度中第一个标记的 pos 标签(通常是 DET 或 ADJ)而不是 NOUN。

代码如下:

def noun_chunk_retokenizer(doc):
with doc.retokenize() as retokenizer:
for chunk in doc.noun_chunks:
retokenizer.merge(chunk)
return doc

nlp = spacy.load('en_core_web_sm')
nlp.add_pipe(noun_chunk_retokenizer)

query = "when is the tennis match happening?"
[(c.text,c.pos_) for c in nlp(query)]

这是我得到的结果:

[('when', 'ADV'),
('is', 'VERB'),
('the tennis match', 'DET'),
('happening', 'VERB'),
('?', 'PUNCT')]

但我希望“网球比赛”被标记为“名词”,这就是它在位移演示中的工作方式:https://explosion.ai/demos/displacy

似乎应该有一个“标准”的方式来做到这一点,但我不确定如何。

最佳答案

你应该使用 built-in merge_noun_chunks component .查看Pipeline Functions documenation :

Merge noun chunks into a single token. Also available via the string name "merge_noun_chunks". After initialization, the component is typically added to the processing pipeline using nlp.add_pipe.

字符串的示例用法:

import spacy
nlp = spacy.load('en_core_web_sm')
nlp.add_pipe(nlp.create_pipe('merge_noun_chunks'))
query = "when is the tennis match happening?"
[(c.text,c.pos_) for c in nlp(query)]

输出:

[('when', 'ADV'),
('is', 'VERB'),
('the tennis match', 'NOUN'),
('happening', 'VERB'),
('?', 'PUNCT')]

关于“源码是怎么做到的”问题请引用spacy Github repo , /spaCy/blob/master/spaCy/pipeline/functions.py 文件第 7 行:

def merge_noun_chunks(doc):
"""Merge noun chunks into a single token.
doc (Doc): The Doc object.
RETURNS (Doc): The Doc object with merged noun chunks.
DOCS: https://spacy.io/api/pipeline-functions#merge_noun_chunks
"""
if not doc.is_parsed:
return doc
with doc.retokenize() as retokenizer:
for np in doc.noun_chunks:
attrs = {"tag": np.root.tag, "dep": np.root.dep}
retokenizer.merge(np, attrs=attrs)
return doc

关于python - 名词短语合并后如何获得句子的正确 pos 标签?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/56599890/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com