gpt4 book ai didi

Keras LSTM - 分类交叉熵降至 0

转载 作者:行者123 更新时间:2023-12-04 17:48:01 25 4
gpt4 key购买 nike

我目前正在尝试比较一些 RNN,但我只有 LSTM 有问题,我不知道为什么。

我正在使用相同的代码/数据集训练 LSTM、SimpleRNN 和 GRU。对于所有这些,损失正常减少。但是对于 LSTM,在某个点之后(损失在 0.4 左右),损失直接下降到 10e-8。如果我尝试预测输出,我只有 Nan。

这是代码:

nb_unit = 7
inp_shape = (maxlen, 7)
loss_ = "categorical_crossentropy"
metrics_ = "categorical_crossentropy"
optimizer_ = "Nadam"
nb_epoch = 250
batch_size = 64

model = Sequential()

model.add(LSTM( units=nb_unit,
input_shape=inp_shape,
return_sequences=True,
activation='softmax')) # I just change the cell name
model.compile(loss=loss_,
optimizer=optimizer_,
metrics=[metrics_])

checkpoint = ModelCheckpoint("lstm_simple.h5",
monitor=loss_,
verbose=1,
save_best_only=True,
save_weights_only=False,
mode='auto',
period=1)
early = EarlyStopping( monitor='loss',
min_delta=0,
patience=10,
verbose=1,
mode='auto')

history = model.fit(X_train, y_train,
validation_data=(X_test, y_test),
epochs=nb_epoch,
batch_size=batch_size,
verbose=2,
callbacks = [checkpoint, early])

这是具有相同输入的 GRU 和 LSTM 的输出:

Input :
[[[1 0 0 0 0 0 0]
[0 1 0 0 0 0 0]
[0 0 0 1 0 0 0]
[0 0 0 1 0 0 0]
[0 1 0 0 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 1 0 0]
[0 0 0 1 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 1 0 0]
[0 0 0 1 0 0 0]
[0 1 0 0 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 1 0 0]
[0 0 0 1 0 0 0]
[0 0 0 0 0 1 0]
[0 0 0 0 0 1 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]]]


LSTM predicts :
[[[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]
[ nan nan nan nan nan nan nan]]]


GRU predicts :
[[[ 0. 0.54 0. 0. 0.407 0. 0. ]
[ 0. 0.005 0.66 0.314 0. 0. 0.001]
[ 0. 0.001 0.032 0.957 0. 0.004 0. ]
[ 0. 0.628 0. 0. 0. 0.372 0. ]
[ 0. 0.555 0. 0. 0. 0.372 0. ]
[ 0. 0. 0. 0. 0.996 0.319 0. ]
[ 0. 0. 0.167 0.55 0. 0. 0. ]
[ 0. 0.486 0. 0.002 0. 0.51 0. ]
[ 0. 0.001 0. 0. 0.992 0.499 0. ]
[ 0. 0. 0.301 0.55 0. 0. 0. ]
[ 0. 0.396 0.001 0.007 0. 0.592 0. ]
[ 0. 0.689 0. 0. 0. 0.592 0. ]
[ 0. 0.001 0. 0. 0.997 0.592 0. ]
[ 0. 0. 0.37 0.55 0. 0. 0. ]
[ 0. 0.327 0.003 0.025 0. 0.599 0. ]
[ 0. 0.001 0. 0. 0.967 0.599 0.002]
[ 0. 0. 0. 0. 0. 0.002 0.874]
[ 0.004 0.076 0.128 0.337 0.02 0.069 0.378]
[ 0.006 0.379 0.047 0.113 0.029 0.284 0.193]
[ 0.006 0.469 0.001 0.037 0.13 0.295 0.193]]]

对于损失,您可以在 fit() 历史的最后几行下面找到:

Epoch 116/250
Epoch 00116: categorical_crossentropy did not improve
- 2s - loss: 0.3774 - categorical_crossentropy: 0.3774 - val_loss: 0.3945 - val_categorical_crossentropy: 0.3945

Epoch 117/250
Epoch 00117: categorical_crossentropy improved from 0.37673 to 0.08198, saving model to lstm_simple.h5
- 2s - loss: 0.0820 - categorical_crossentropy: 0.0820 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08

Epoch 118/250
Epoch 00118: categorical_crossentropy improved from 0.08198 to 0.00000, saving model to lstm_simple.h5
- 2s - loss: 7.5460e-08 - categorical_crossentropy: 7.5460e-08 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08

或者损失基于Epochs的演化。

enter image description here

我之前在没有 Softmax 和 MSE 作为损失函数的情况下尝试过,我没有得到任何错误。

如果需要,您可以在 Github (https://github.com/Coni63/SO/blob/master/Reber.ipynb) 上找到用于生成数据集的笔记本和脚本。

非常感谢您的支持,问候,尼古拉斯

编辑 1:

根本原因似乎是 Softmax 函数消失了。如果我在它崩溃之前停止它并显示我拥有的每个时间步长的 softmax 总和:

LSTM :
[[ 0.112]
[ 0.008]
[ 0.379]
[ 0.04 ]
[ 0.001]
[ 0.104]
[ 0.021]
[ 0. ]
[ 0.104]
[ 0.343]
[ 0.012]
[ 0. ]
[ 0.23 ]
[ 0.13 ]
[ 0.147]
[ 0.145]
[ 0.152]
[ 0.157]
[ 0.163]
[ 0.169]]


GRU :
[[ 0.974]
[ 0.807]
[ 0.719]
[ 1.184]
[ 0.944]
[ 0.999]
[ 1.426]
[ 0.957]
[ 0.999]
[ 1.212]
[ 1.52 ]
[ 0.954]
[ 0.42 ]
[ 0.83 ]
[ 0.903]
[ 0.944]
[ 0.976]
[ 1.005]
[ 1.022]
[ 1.029]]

Softmax 为 0,下一步将尝试除以 0。现在我不知道如何修复它。

最佳答案

我只是发布我当前的解决方案,以防其他人将来遇到此问题。

为了避免消失,我添加了一个简单的全连接层,其输出大小与输入相同,之后它就可以正常工作了。该层允许对 LSTM/GRU/SRNN 的输出进行另一种“配置”,并避免输出消失。

这是最终代码:

nb_unit = 7
inp_shape = (maxlen, 7)
loss_ = "categorical_crossentropy"
metrics_ = "categorical_crossentropy"
optimizer_ = "Nadam"
nb_epoch = 250
batch_size = 64

model = Sequential()

model.add(LSTM(units=nb_unit,
input_shape=inp_shape,
return_sequences=True)) # LSTG/GRU/SimpleRNN
model.add(Dense(7, activation='softmax')) # New
model.compile(loss=loss_,
optimizer=optimizer_,
metrics=[metrics_])

checkpoint = ModelCheckpoint("lstm_simple.h5",
monitor=loss_,
verbose=1,
save_best_only=True,
save_weights_only=False,
mode='auto',
period=1)
early = EarlyStopping(
monitor='loss',
min_delta=0,
patience=10,
verbose=1,
mode='auto')

我希望这可以帮助其他人:)

关于Keras LSTM - 分类交叉熵降至 0,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/47460383/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com