tensorflow - Huggingface 微调——如何在预训练的基础上构建自定义模型

Question

问题

请帮助理解以下问题的原因以及如何构建 Keras 模型以在 huggingface 的预训练模型之上进行微调。

目标

在 TFDistilBertForSequenceClassification 之上为 DistilBERT 微调创建自定义模型来自 Huggingface。

为模型输入形状

根据分词器输出的形状，我假设它是 (2, None, 256)作为[input_ids, attention_mask]将进入模型。

分词器的输出。

from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')

max_sequence_length = 256
tokens = tokenizer(
    " ".join(["token"] * max_sequence_length), 
    truncation=True,
    padding=True,
    max_length=max_sequence_length,
    return_tensors="tf"
)
print(tokens)
---
{
  'input_ids':      <tf.Tensor: shape=(1, 256), dtype=int32, numpy=array([[  101, 19204, 19204, 19204, 19204, 19204, 19204, 19204, 19204, ...]], dtype=int32)>, 
  'attention_mask': <tf.Tensor: shape=(1, 256), dtype=int32, numpy=array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...]], dtype=int32)>
}

预训练模型

model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
for layer in model.layers:
    if layer.name == "distilbert":
        layer.trainable = False
model.summary()
---
Model: "tf_distil_bert_for_sequence_classification_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
distilbert (TFDistilBertMain multiple                  66362880  
_________________________________________________________________
pre_classifier (Dense)       multiple                  590592    
_________________________________________________________________
classifier (Dense)           multiple                  1538      
_________________________________________________________________
dropout_99 (Dropout)         multiple                  0         
=================================================================
Total params: 66,955,010
Trainable params: 592,130
Non-trainable params: 66,362,880

自定义模型

使用 Sequential 在预训练模型之上添加了一个 Keras Dense 层。

seq = Sequential([
   model,
   Dense(
       name="output_softmax", 
       units=2, 
       activation="softmax"
   )
])
seq.compile(
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=tf.keras.optimizers.Adam()
)

问题

The base Layer class说构建方法创建权重。

build(self, input_shape): This method can be used to create weights that depend on the shape(s) of the input(s), using add_weight(). call() will automatically build the layer (if it has not been built yet) by calling build().

运行该方法但出现错误。

seq.build(input_shape=(2, None, max_sequence_length))
---
...
ValueError: You cannot build your model by calling `build` if your layers do not support float-type inputs. Instead, in order to instantiate and build your model, `call` your model on real tensor data (of the correct type).

根据错误消息，将分词器输出提供给模型并得到另一个错误。

seq(tokens)
---
TypeError: Failed to convert 'TFSequenceClassifierOutput(loss=None, logits=TensorShape([1, 2]), hidden_states=None, attentions=None)' to a shape: ''logits''could not be converted to a dimension. A shape should either be single dimension (e.g. 10), or an iterable of dimensions (e.g. [1, 10, None]).

环境

python --version
---
Python 3.7.10

print(tf.__version__)
---
2.5.0

print(transformers.__version__)
---
4.8.2

Answer 1

在不使用 Sequential 或 build 方法的情况下，您尝试过 Keras Functional API 吗？-(我尝试过并使用其他预训练模型)s.a:-(伪代码)

def custom():
   x=Input(shape=(256,))
   y=Input(shape=(256,))
   out=distilbertlayer([x,y])
   out=Dense(2,activation='softmax')(out)
   mod=tf.keras.models.Model([x,y],out)
   return mod

custommodel=custom()

在这里，在给定的信息中。我认为错误是由于传递给自定义密集层的输出类型错误。作为建议，您可以尝试将不同的输出从 distilbertlayer 传递到自定义密集层，例如

out=distilbertlayer([x,y])
out=Dense(2,activation='softmax')(out[:,0])

但是，首先应该了解 distilberlayer 的输出格式。