python - Keras 分类器的准确度在训练期间稳步上升，然后下降到 0.25(局部最小值？)

Question

我有以下神经网络，使用 Tensorflow 作为后端用 Keras 编写，我在 Windows 10 上的 Python 3.5 (Anaconda) 上运行:

    model = Sequential() 
    model.add(Dense(100, input_dim=283, init='normal', activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(150, init='normal', activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(200, init='normal', activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(200, init='normal', activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(200, init='normal', activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(4, init='normal', activation='sigmoid'))
    sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

我正在我的 GPU 上训练。在训练期间(10000 个纪元)，朴素网络的准确度从 0.25 稳步增加到 0.7 到 0.9 之间的某个位置，然后突然下降并停留在 0.25:

    Epoch 1/10000
    6120/6120 [==============================] - 1s - loss: 1.5329 - acc: 0.2665
    Epoch 2/10000
    6120/6120 [==============================] - 1s - loss: 1.2985 - acc: 0.3784
    Epoch 3/10000
    6120/6120 [==============================] - 1s - loss: 1.2259 - acc: 0.4891
    Epoch 4/10000
    6120/6120 [==============================] - 1s - loss: 1.1867 - acc: 0.5208
    Epoch 5/10000
    6120/6120 [==============================] - 1s - loss: 1.1494 - acc: 0.5199
    Epoch 6/10000
    6120/6120 [==============================] - 1s - loss: 1.1042 - acc: 0.4953
    Epoch 7/10000
    6120/6120 [==============================] - 1s - loss: 1.0491 - acc: 0.4982
    Epoch 8/10000
    6120/6120 [==============================] - 1s - loss: 1.0066 - acc: 0.5065
    Epoch 9/10000
    6120/6120 [==============================] - 1s - loss: 0.9749 - acc: 0.5338
    Epoch 10/10000
    6120/6120 [==============================] - 1s - loss: 0.9456 - acc: 0.5696
    Epoch 11/10000
    6120/6120 [==============================] - 1s - loss: 0.9252 - acc: 0.5995
    Epoch 12/10000
    6120/6120 [==============================] - 1s - loss: 0.9111 - acc: 0.6106
    Epoch 13/10000
    6120/6120 [==============================] - 1s - loss: 0.8772 - acc: 0.6160
    Epoch 14/10000
    6120/6120 [==============================] - 1s - loss: 0.8517 - acc: 0.6245
    Epoch 15/10000
    6120/6120 [==============================] - 1s - loss: 0.8170 - acc: 0.6345
    Epoch 16/10000
    6120/6120 [==============================] - 1s - loss: 0.7850 - acc: 0.6428
    Epoch 17/10000
    6120/6120 [==============================] - 1s - loss: 0.7633 - acc: 0.6580
    Epoch 18/10000
    6120/6120 [==============================] - 4s - loss: 0.7375 - acc: 0.6717
    Epoch 19/10000
    6120/6120 [==============================] - 1s - loss: 0.7058 - acc: 0.6850
    Epoch 20/10000
    6120/6120 [==============================] - 1s - loss: 0.6787 - acc: 0.7018
    Epoch 21/10000
    6120/6120 [==============================] - 1s - loss: 0.6557 - acc: 0.7093
    Epoch 22/10000
    6120/6120 [==============================] - 1s - loss: 0.6304 - acc: 0.7208
    Epoch 23/10000
    6120/6120 [==============================] - 1s - loss: 0.6052 - acc: 0.7270
    Epoch 24/10000
    6120/6120 [==============================] - 1s - loss: 0.5848 - acc: 0.7371
    Epoch 25/10000
    6120/6120 [==============================] - 1s - loss: 0.5564 - acc: 0.7536
    Epoch 26/10000
    6120/6120 [==============================] - 1s - loss: 0.1787 - acc: 0.4163
    Epoch 27/10000
    6120/6120 [==============================] - 1s - loss: 1.1921e-07 - acc: 0.2500
    Epoch 28/10000
    6120/6120 [==============================] - 1s - loss: 1.1921e-07 - acc: 0.2500
    Epoch 29/10000
    6120/6120 [==============================] - 1s - loss: 1.1921e-07 - acc: 0.2500
    Epoch 30/10000
    6120/6120 [==============================] - 2s - loss: 1.1921e-07 - acc: 0.2500
    Epoch 31/10000
    6120/6120 [==============================] - 1s - loss: 1.1921e-07 - acc: 0.2500
    Epoch 32/10000
    6120/6120 [==============================] - 1s - loss: 1.1921e-07 - acc: 0.2500 ...

我猜这是由于优化器陷入局部最小值，它将所有数据分配给一个类别。我怎样才能阻止它这样做？

我尝试过的事情(但似乎并没有阻止这种情况发生):

1. 使用不同的优化器 (adam)
2. 确保训练数据包含来自每个类别的相同数量的示例
3. 增加训练数据量(目前为 6000)
4. 在 2 到 5 之间改变类别数
5. 将网络中的隐藏层数从 1 层增加到 5 层
6. 更改层的宽度(从 50 到 500)

这些都没有帮助。为什么会发生这种情况和/或如何抑制它的任何其他想法？这可能是 Keras 中的错误吗？非常感谢您提出任何建议。

编辑:通过将最终激活更改为 softmax(来自 sigmoid)并将 maxnorm(3) 正则化添加到最后两个隐藏层，问题似乎已经解决:

model = Sequential() 
model.add(Dense(100, input_dim=npoints, init='normal', activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(150, init='normal', activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(200, init='normal', activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(200, init='normal', activation='relu', W_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(200, init='normal', activation='relu', W_constraint=maxnorm(3)))
model.add(Dropout(0.2))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.add(Dense(ncat, init='normal', activation='softmax'))
model.compile(loss='mean_squared_error', optimizer=sgd, metrics=['accuracy'])

非常感谢您的建议。

Answer 1

问题出在 sigmoid 函数作为最后一层的激活。在这种情况下，您最后一层的输出不能解释为属于单个类的给定示例的概率分布。该层的输出通常甚至不等于 1。在这种情况下，优化可能会导致意外行为。在我看来，添加 maxnorm 约束不是必需的，但我强烈建议您使用 categorical_crossentropy 而不是 mse 损失，因为已证明此函数更适合这种优化案例。