python - 简单的 Keras 神经网络不学习

Question

我正在尝试复制 Neural Networks and Deep Learning 中的一些示例使用 Keras，但我在基于第 1 章的体系结构训练网络时遇到问题。目的是对 MNIST 数据集中的书面数字进行分类。架构:

• 784 个输入(MNIST 图像中每个 28 * 28 像素一个)
• 30 个神经元的隐藏层
• 10 个神经元的输出层
• 权重和偏差根据均值为 0 且标准差为 1 的高斯分布进行初始化。
• 损失/成本函数是均方误差。
• 优化器是随机梯度下降。

超参数:

• 学习率 = 3.0
• 批量大小 = 10
• 时代 = 30

我的代码:

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.initializers import RandomNormal


# import data
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# input image dimensions
img_rows, img_cols = 28, 28

x_train = x_train.reshape(x_train.shape[0], img_rows * img_cols)
x_test = x_test.reshape(x_test.shape[0], img_rows * img_cols)
input_shape = (img_rows * img_cols,)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
num_classes = 10
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
print('y_train shape:', y_train.shape)

# Construct model
# 784 * 30 * 10
# Normal distribution for weights/biases
# Stochastic Gradient Descent optimizer
# Mean squared error loss (cost function)
model = Sequential()
layer1 = Dense(30,
               input_shape=input_shape,
               kernel_initializer=RandomNormal(stddev=1),
               bias_initializer=RandomNormal(stddev=1))
model.add(layer1)
layer2 = Dense(10,
               kernel_initializer=RandomNormal(stddev=1),
               bias_initializer=RandomNormal(stddev=1))
model.add(layer2)
print('Layer 1 input shape: ', layer1.input_shape)
print('Layer 1 output shape: ', layer1.output_shape)
print('Layer 2 input shape: ', layer2.input_shape)
print('Layer 2 output shape: ', layer2.output_shape)

model.summary()
model.compile(optimizer=SGD(lr=3.0),
              loss='mean_squared_error',
              metrics=['accuracy'])

# Train 
model.fit(x_train,
          y_train,
          batch_size=10,
          epochs=30,
          verbose=2)

# Run on test data and output results
result = model.evaluate(x_test,
                        y_test,
                        verbose=1)
print('Test loss: ', result[0])
print('Test accuracy: ', result[1])

输出(使用 Python 3.6 和 TensorFlow 后端):

Using TensorFlow backend.
x_train shape: (60000, 784)
60000 train samples
10000 test samples
y_train shape: (60000, 10)
Layer 1 input shape:  (None, 784)
Layer 1 output shape:  (None, 30)
Layer 2 input shape:  (None, 30)
Layer 2 output shape:  (None, 10)
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 30)                23550     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                310       
=================================================================
Total params: 23,860
Trainable params: 23,860
Non-trainable params: 0
_________________________________________________________________
Epoch 1/30
 - 7s - loss: nan - acc: 0.0987
Epoch 2/30
 - 7s - loss: nan - acc: 0.0987

(对所有 30 个 epoch 重复)

Epoch 30/30
 - 6s - loss: nan - acc: 0.0987
10000/10000 [==============================] - 0s 22us/step
Test loss:  nan
Test accuracy:  0.098

如您所见，网络根本没有学习，我不确定为什么。据我所知，这些形状看起来不错。我在做什么会阻止网络学习？

(顺便说一句，我知道交叉熵损失和 softmax 输出层会更好；但是，从链接的书来看，它们似乎不是必需的。本书第 1 章中手动实现的网络学习成功；我在继续之前尝试复制它。)

Answer 1

您需要指定每一层的激活。所以对于每一层。应该是这样的:

layer2 = Dense(10,
           activation='sigmoid',
           kernel_initializer=RandomNormal(stddev=1),
           bias_initializer=RandomNormal(stddev=1))

注意我在这里指定了激活参数。同样对于最后一层，您应该使用 activation="softmax" 因为您有多个类别。

另一件需要考虑的事情是，分类(与回归相反)在熵损失的情况下效果最好。因此，您可能希望将 model.compile 中的损失值更改为 loss='categorical_crossentropy'。但是，这不是必需的，您仍然会使用 mean_square_error 损失得到结果。

如果您仍然得到 nan 损失值，您可以尝试更改 SGD 的学习率。

我使用您显示的脚本获得了 0.9425 的测试精度，只需将第一层的激活更改为 sigmoid 并将第二层的激活更改为 softmax.