python - 多层前馈网络无法在 TensorFlow 中训练

Question

我从 TensorFlow 教程开始，使用单层前馈神经网络对 mnist 数据集中的图像进行分类。效果很好，我在测试集上得到了 80% 以上的成绩。然后我尝试通过在中间添加一个新层来将其修改为多层网络。修改后，我所有训练网络的尝试都失败了。前几次迭代网络变得更好一些，但随后它的准确率停滞在 11.35%。

使用 1 个隐藏层的前 20 次迭代:

Train set: 0.124, test set: 0.098
Train set: 0.102, test set: 0.098
Train set: 0.112, test set: 0.101
Train set: 0.104, test set: 0.101
Train set: 0.092, test set: 0.101
Train set: 0.128, test set: 0.1135
Train set: 0.12, test set: 0.1135
Train set: 0.114, test set: 0.1135
Train set: 0.108, test set: 0.1135
Train set: 0.1, test set: 0.1135
Train set: 0.114, test set: 0.1135
Train set: 0.11, test set: 0.1135
Train set: 0.122, test set: 0.1135
Train set: 0.102, test set: 0.1135
Train set: 0.12, test set: 0.1135
Train set: 0.106, test set: 0.1135
Train set: 0.102, test set: 0.1135
Train set: 0.116, test set: 0.1135
Train set: 0.11, test set: 0.1135
Train set: 0.124, test set: 0.1135

不管我训练它多久，它都卡在这里。我尝试从修正线性单位更改为 softmax，两者都会产生相同的结果。我尝试将适应度函数更改为 e=(y_true-y)^2。结果相同。

不使用隐藏层的前二十次迭代:

Train set: 0.124, test set: 0.098
Train set: 0.374, test set: 0.3841
Train set: 0.532, test set: 0.5148
Train set: 0.7, test set: 0.6469
Train set: 0.746, test set: 0.7732
Train set: 0.786, test set: 0.8
Train set: 0.788, test set: 0.7887
Train set: 0.752, test set: 0.7882
Train set: 0.84, test set: 0.8138
Train set: 0.85, test set: 0.8347
Train set: 0.806, test set: 0.8084
Train set: 0.818, test set: 0.7917
Train set: 0.85, test set: 0.8063
Train set: 0.792, test set: 0.8268
Train set: 0.812, test set: 0.8259
Train set: 0.774, test set: 0.8053
Train set: 0.788, test set: 0.8522
Train set: 0.812, test set: 0.8131
Train set: 0.814, test set: 0.8638
Train set: 0.778, test set: 0.8604

这是我的代码:

import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Parameters
batch_size = 500

# Create the network structure
# ----------------------------

# First layer
x = tf.placeholder(tf.float32, [None, 784])
W_1 = tf.Variable(tf.zeros([784,10]))
b_1 = tf.Variable(tf.zeros([10]))
y_1 = tf.nn.relu(tf.matmul(x,W_1) + b_1)

# Second layer
W_2 = tf.Variable(tf.zeros([10,10]))
b_2 = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(y_1,W_2) + b_2)

# Loss function
y_true = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y), reduction_indices=[1]))

# Training method
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Train network
# -------------
sess = tf.Session()
sess.run(tf.initialize_all_variables())
batch, batch_labels = mnist.train.next_batch(batch_size)
for i in range(20):
    print("Train set: " + str(sess.run(accuracy, feed_dict={x: batch, y_true: batch_labels}))
            + ", test set: " + str(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))) 
    sess.run(train_step, feed_dict={x: batch, y_true: batch_labels})
    batch, batch_labels = mnist.train.next_batch(batch_size)

所以用这个代码它不起作用，但如果我改变

y = tf.nn.softmax(tf.matmul(y_1,W_2) + b_2)

至

y = tf.nn.softmax(tf.matmul(x,W_1) + b_1)

然后就可以了。我错过了什么？

编辑:现在我可以正常工作了。需要进行两个更改，首先将权重初始化为随机值而不是零(是的，实际上权重需要不为零，尽管有 relu 函数，但偏差为零是可以的)。第二件事对我来说很奇怪:如果我从输出层中删除 softmax 函数，而不是手动应用交叉熵公式，而是使用 softmax_cross_entropy_with_logits(y,y_true) 函数，那么它就可以工作。据我了解，应该是相同的..之前我也尝试过误差平方和，但也不起作用..无论如何，下面的代码正在工作。 (虽然相当丑陋，但工作......)经过 10k 次迭代，它在测试集上的准确率达到了 93.59%，所以无论如何都不是最佳的，但比没有隐藏层的要好。仅 20 次迭代后，它已经达到 65%。

import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Parameters
batch_size = 500

# Create the network structure
# ----------------------------

# First layer
x = tf.placeholder(tf.float32, [None, 784])
W_1 = tf.Variable(tf.truncated_normal([784,10], stddev=0.1))
b_1 = tf.Variable(tf.truncated_normal([10], stddev=0.1))
y_1 = tf.nn.relu(tf.matmul(x,W_1) + b_1)

# Second layer
W_2 = tf.Variable(tf.truncated_normal([10,10], stddev=0.1))
b_2 = tf.Variable(tf.truncated_normal([10], stddev=0.1))
y = tf.matmul(y_1,W_2) + b_2

# Loss function
y_true = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y,y_true))

# Training method
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Train network
# -------------
sess = tf.Session()
sess.run(tf.initialize_all_variables())
batch, batch_labels = mnist.train.next_batch(batch_size)
for i in range(10000):
    if i % 100 == 0:
        print("Train set: " + str(sess.run(accuracy, feed_dict={x: batch, y_true: batch_labels}))
                + ", test set: " + str(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))) 
    sess.run(train_step, feed_dict={x: batch, y_true: batch_labels})
    batch, batch_labels = mnist.train.next_batch(batch_size)

Answer 1

一些建议:

1-为两个权重变量初始化添加标准差，而不是使用零进行初始化:

weight_1 = tf.Variable(tf.truncated_normal([784,10], stddev=0.1))

2-降低学习率，直到准确度值显示出不同的行为。

3- 使用 RELU 时，用稍微正的值初始化偏差。此建议可能与您遇到的问题关系不大。

bias_1 = tf.Variable(tf.constant(.05, shape=[10]))