nlp - 将NER SpaCy格式转换为IOB格式

Question

我有已经以 SpaCy 格式标记的数据。例如:

("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
("I like London and Berlin.", {"entities": [(7, 13, "LOC"), (18, 24, "LOC")]})

但我想尝试使用任何其他 NER 模型来训练它，例如 BERT-NER，它需要 IOB 标记。是否有从 SpaCy 数据格式到 IOB 的转换代码？

谢谢!

Answer 1

这与 https://stackoverflow.com/a/59209377/461847 密切相关并且大部分是从这里复制的。 ，请参阅那里的评论中的注释:

import spacy
from spacy.gold import biluo_tags_from_offsets

TRAIN_DATA = [
    ("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
    ("I like London and Berlin.", {"entities": [(7, 13, "LOC"), (18, 24, "LOC")]}),
]

nlp = spacy.load('en_core_web_sm')
docs = []
for text, annot in TRAIN_DATA:
    doc = nlp(text)
    tags = biluo_tags_from_offsets(doc, annot['entities'])
    # then convert L->I and U->B to have IOB tags for the tokens in the doc