python - Keras Transfer Learning Resnet50 using fit_generator got high acc but low val_acc 问题

Question

我正在使用 Resnet50 模型进行迁移学习，总共使用 20 个场景中的 100,000 张图像(MIT Place365 数据集)。我只训练了最后 160 层(由于内存限制)。问题是我得到了相当高的准确性但验证准确性极低，我认为这可能是一个过度拟合的问题但我不知道如何解决它。如果有人能就如何解决我的低 val_acc 问题给我建议，我将不胜感激，非常感谢。我的代码如下:

V1 = np.load("C:/Users/Desktop/numpydataKeras_20_val/imgonehot_val_500.npy")
V2 = np.load("C:/Users/Desktop/numpydataKeras_20_val/labelonehot_val_500.npy") 


net = keras.applications.resnet50.ResNet50(include_top=False, weights='imagenet', input_tensor=None, input_shape=(224, 224, 3))

x = net.output
x = Flatten()(x)
x = Dense(128)(x)
x = Activation('relu')(x)
x = Dropout(0.5)(x)
output_layer = Dense(20, activation='softmax', name='softmax')(x)
net_final = Model(inputs=net.input, outputs=output_layer)

for layer in net_final.layers[:-160]:
    layer.trainable = False
for layer in net_final.layers[-160:]:
    layer.trainable = True

net_final.compile(Adam(lr=.00002122), loss='categorical_crossentropy', metrics=['accuracy'])

def data_generator():
    n = 100000
    Num_batch = 100000/100
    arr = np.arange(1000)
    np.random.shuffle(arr)
    while (True):
        for i in arr:
            seed01 = random.randint(0,1000000)

            X_batch  = np.load( "C:/Users/Desktop/numpydataKeras/imgonehot_"+str((i+1)*100)+".npy" )
            np.random.seed(seed01)
            np.random.shuffle(X_batch)

            y_batch = np.load( "C:/Users/Desktop/numpydataKeras/labelonehot_"+str((i+1)*100)+".npy" )
            np.random.seed(seed01)
            np.random.shuffle(y_batch)

            yield X_batch, y_batch

weights_file = 'C:/Users/Desktop/Transfer_learning_resnet50_fit_generator_02s.h5'
early_stopping = EarlyStopping(monitor='val_acc', patience=5, mode='auto', verbose=2)
model_checkpoint = ModelCheckpoint(weights_file, monitor='val_acc', save_best_only=True, verbose=2)
callbacks = [early_stopping, model_checkpoint]

model_fit = net_final.fit_generator(
    data_generator(),
    steps_per_epoch=1000,
    epochs=5,
    validation_data=(V1, V2),
    callbacks=callbacks,
    verbose=1,
    pickle_safe=False)

打印结果如下:

Epoch 1/5
1000/1000 [==============================] - 3481s 3s/step - loss: 1.7917 - acc: 0.4757 - val_loss: 3.5872 - val_acc: 0.0560

Epoch 00001: val_acc improved from -inf to 0.05600, saving model to C:/Users/Desktop/Transfer_learning_resnet50_fit_generator_02s.h5
Epoch 2/5
1000/1000 [==============================] - 4884s 5s/step - loss: 1.1287 - acc: 0.6595 - val_loss: 4.2113 - val_acc: 0.0520

Epoch 00002: val_acc did not improve from 0.05600
Epoch 3/5
1000/1000 [==============================] - 4964s 5s/step - loss: 0.8033 - acc: 0.7464 - val_loss: 4.9595 - val_acc: 0.0520

Epoch 00003: val_acc did not improve from 0.05600
Epoch 4/5
1000/1000 [==============================] - 4961s 5s/step - loss: 0.5677 - acc: 0.8143 - val_loss: 4.5484 - val_acc: 0.0520

Epoch 00004: val_acc did not improve from 0.05600
Epoch 5/5
1000/1000 [==============================] - 4928s 5s/step - loss: 0.3999 - acc: 0.8672 - val_loss: 4.6155 - val_acc: 0.0400

Epoch 00005: val_acc did not improve from 0.05600

Answer 1

正在关注 https://github.com/keras-team/keras/issues/9214#issuecomment-397916155看来批量归一化应该是可训练的。

以下代码可以替换您设置/取消设置可训练层的循环:

for layer in model.layers:
    if hasattr(layer, 'moving_mean') and hasattr(layer, 'moving_variance'):
        layer.trainable = True
        K.eval(K.update(layer.moving_mean, K.zeros_like(layer.moving_mean)))
        K.eval(K.update(layer.moving_variance, K.zeros_like(layer.moving_variance)))
    else:
        layer.trainable = False

在我自己的数据上，我需要减少批量大小以避免 OOM，我现在有:

Epoch 1/10
470/470 [==============================] - 90s 192ms/step - loss: 0.3513 - acc: 0.8660 - val_loss: 0.1299 - val_acc: 0.9590
Epoch 2/10
470/470 [==============================] - 83s 177ms/step - loss: 0.2204 - acc: 0.9163 - val_loss: 0.1276 - val_acc: 0.9471
Epoch 3/10
470/470 [==============================] - 83s 177ms/step - loss: 0.2219 - acc: 0.9184 - val_loss: 0.1048 - val_acc: 0.9589
Epoch 4/10
470/470 [==============================] - 83s 177ms/step - loss: 0.1813 - acc: 0.9327 - val_loss: 0.1857 - val_acc: 0.9303

警告，这可能会影响准确性，您必须卡住模型以避免出现奇怪的推论。但这似乎是对我有用的唯一方法。

另一条评论 https://github.com/keras-team/keras/issues/9214#issuecomment-422490253如果它是批量归一化，则只检查图层名称以将其设置为可训练，但它对我来说没有任何改变。也许它可以帮助您的数据集。