gpt4 book ai didi

keras - 如何将 Bert 嵌入提供给 LSTM

转载 作者:行者123 更新时间:2023-12-03 19:30:06 25 4
gpt4 key购买 nike

我正在研究用于文本分类问题的 Bert + MLP 模型。本质上,我试图用基本的 LSTM 模型替换 MLP 模型。

是否可以创建带有嵌入的 LSTM?或者,最好创建一个带有嵌入层的 LSTM?

更具体地说,我很难尝试创建嵌入矩阵,因此我可以使用 Bert 嵌入创建嵌入层。

def get_bert_embeddings(dataset='gap_corrected_train',
dataset_path=TRAIN_PATH,
bert_path=BERT_UNCASED_LARGE_PATH,
bert_layers=BERT_LAYERS):
"""Get BERT embeddings for all files in dataset_path and specified BERT layers and write them to file."""
df = None
for file in os.listdir(dataset_path):
if df is None:
df = pd.read_csv(dataset_path+'/'+file, sep='\t')
else:
next_df = pd.read_csv(dataset_path+'/'+file, sep='\t')
df = pd.concat([df, next_df], axis=0)
df.reset_index(inplace=True, drop=True)

for i, layer in enumerate(bert_layers):
embeddings_file = INTERIM_PATH + 'emb_bert' + str(layer) + '_' + dataset + '.h5'
if not os.path.exists(embeddings_file):
print('Embeddings file: ', embeddings_file)
print('Extracting BERT Layer {0} embeddings for {1}...'.format(layer, dataset))
print("Started at ", time.ctime())

emb = get_bert_token_embeddings(df, bert_path, layer)
emb.to_hdf(embeddings_file, 'table')

print("Finished at ", time.ctime())

def build_mlp_model(input_shape):
input_layer = layers.Input(input_shape)



input_features = layers.Input((len(FEATURES),))
x = layers.Concatenate(axis=1, name="concate_layer")([input_layer, input_features])


x = layers.Dense(HIDDEN_SIZE, name='dense1')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.Dropout(DROPOUT, seed=RANDOM)(x)

x = layers.Dense(HIDDEN_SIZE//2, name='dense2')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.Dropout(DROPOUT//2, seed=RANDOM)(x)

x = layers.Dense(HIDDEN_SIZE//4, name='dense3')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.Dropout(DROPOUT//2, seed=RANDOM)(x)

output_layer = layers.Dense(3, name='output', kernel_regularizer = regularizers.l2(LAMBDA))(x)
output_layer = layers.Activation('softmax')(output_layer)

model = models.Model(input=[input_layer, input_features], output=output_layer, name="mlp")
return model

最佳答案

您可以创建首先使用 Embedding 层,然后是 LSTM 然后是 Dense 的模型。
比如这里:

deep_inputs = Input(shape=(length_of_your_data,))
embedding_layer = Embedding(vocab_size, output_dim = 3000, trainable=True)(deep_inputs)
LSTM_Layer_1 = LSTM(512)(embedding_layer)
dense_layer_1 = Dense(number_of_classes, activation='softmax')(LSTM_Layer_1)
model_AdGroups = Model(inputs=deep_inputs, outputs=dense_layer_1)

关于keras - 如何将 Bert 嵌入提供给 LSTM,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/55669695/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com