encoding - 如何获得 RoBERTa 词嵌入？

Question

给定一个“Roberta 是 BERT 的高度优化版本”类型的句子，我需要使用 RoBERTa 获取这句话中每个单词的嵌入。我试图在网上查看示例代码，但没有找到明确的答案。

我的看法如下:

tokens = roberta.encode(headline)
all_layers = roberta.extract_features(tokens, return_all_hiddens=True)
embedding = all_layers[0]
n = embedding.size()[1] - 1
embedding = embedding[:,1:n,:]

哪里 embedding[:,1:n,:]用于仅提取句子中单词的嵌入，没有开始和结束标记。

这是正确的吗？

Answer 1

TOKENIZER_PATH = "../input/roberta-transformers-pytorch/roberta-base"
ROBERTA_PATH = "../input/roberta-transformers-pytorch/roberta-base"

text= "How are you? I am good."
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH)

##how the words are broken into tokens
print(tokenizer.tokenize(text))

##the format of a encoding
print(tokenizer.batch_encode_plus([text]))

##op wants the input id
print(tokenizer.batch_encode_plus([text])['input_ids'])

##op wants the input id without first and last token
print(tokenizer.batch_encode_plus([text])['input_ids'][0][1:-1])

输出:
['How', 'Ġare', 'Ġyou', '?', 'ĠI', 'Ġam', 'Ġgood', '.']
{'input_ids': [[0, 6179, 32, 47, 116, 38, 524, 205, 4, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1 , 1, 1]]}
[[0, 6179, 32,47, 116, 38, 524, 205, 4, 2]]
[6179, 32, 47, 116, 38, 524, 205, 4]
不要担心“Ġ”字符。它只是表示单词前面有一个空格。