python - 更改输入数据时 Tensorflow 模型不进行训练

Question

我正在尝试使用 tensorflow 来训练使用交通标志图像的神经网络(LeNet)。我想检查预处理技术对神经网络性能的影响。因此，我对图像进行了预处理，并将结果(训练图像、验证图像、测试图像、最终测试图像)作为元组存储在字典中。

然后我尝试迭代这个字典，然后使用 tensorflow 的训练和验证操作，如下所示

import tensorflow as tf
from sklearn.utils import shuffle


output_data = []
EPOCHS = 5
BATCH_SIZE = 128
rate = 0.0005

for key in finalInputdata.keys():
    for procTypes in range(0,(len(finalInputdata[key]))):
        if np.shape(finalInputdata[key][procTypes][0]) != ():
            X_train = finalInputdata[key][procTypes][0]
            X_valid = finalInputdata[key][procTypes][1]
            X_test = finalInputdata[key][procTypes][2]
            X_finaltest = finalInputdata[key][procTypes][3]


            x = tf.placeholder(tf.float32, (None, 32, 32,np.shape(X_train)[-1]))
            y = tf.placeholder(tf.int32, (None))
            one_hot_y = tf.one_hot(y,43)

            # Tensor Operations
            logits = LeNet(x,np.shape(X_train)[-1])

            cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits,one_hot_y)
            softmax_probability = tf.nn.softmax(logits)

            loss_operation = tf.reduce_mean(cross_entropy)
            optimizer = tf.train.AdamOptimizer(learning_rate=rate)
            training_operation = optimizer.minimize(loss_operation)
            correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(one_hot_y,1))
            accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


                # Pipeline for training and evaluation

            sess = tf.InteractiveSession()

            sess.run(tf.global_variables_initializer())
            num_examples = len(X_train)

            print("Training on %s images processed as %s" %(key,dict_fornames['proctypes'][procTypes]))
            print()
            for i in range(EPOCHS):
                X_train, y_train = shuffle(X_train, y_train)
                for offset in range(0, num_examples, BATCH_SIZE):
                    end = offset + BATCH_SIZE
                    batch_x, batch_y = X_train[offset:end], y_train[offset:end]
                    sess.run(training_operation, feed_dict = {x: batch_x, y: batch_y})

                training_accuracy = evaluate(X_train,y_train)

                validation_accuracy = evaluate(X_valid, y_valid)

                testing_accuracy = evaluate(X_test, y_test)

                final_accuracy = evaluate(X_finaltest, y_finalTest)

                print("EPOCH {} ...".format(i+1))
                print("Training Accuracy = {:.3f}".format(training_accuracy))
                print("Validation Accuracy = {:.3f}".format(validation_accuracy))
                print()
                output_data.append({'EPOCHS':EPOCHS, 'LearningRate':rate, 'ImageType': 'RGB',\
                                    'PreprocType': dict_fornames['proctypes'][0],\
                                    'TrainingAccuracy':training_accuracy, 'ValidationAccuracy':validation_accuracy, \
                                    'TestingAccuracy': testing_accuracy})


            sess.close()

评估函数如下

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0,num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict = {x:batch_x, y:batch_y})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

一旦我执行该程序，它对于数据集的第一次迭代效果很好，但从第二次迭代开始，网络不再训练，并继续对所有其他迭代进行训练。

Training on RGB images processed as Original

EPOCH 1 ...
Training Accuracy = 0.525
Validation Accuracy = 0.474

EPOCH 2 ...
Training Accuracy = 0.763
Validation Accuracy = 0.682

EPOCH 3 ...
Training Accuracy = 0.844
Validation Accuracy = 0.723

EPOCH 4 ...
Training Accuracy = 0.888
Validation Accuracy = 0.779

EPOCH 5 ...
Training Accuracy = 0.913
Validation Accuracy = 0.795

Training on RGB images processed as Mean Subtracted Data

EPOCH 1 ...
Training Accuracy = 0.056
Validation Accuracy = 0.057

EPOCH 2 ...
Training Accuracy = 0.057
Validation Accuracy = 0.057

EPOCH 3 ...
Training Accuracy = 0.057
Validation Accuracy = 0.056

EPOCH 4 ...
Training Accuracy = 0.058
Validation Accuracy = 0.056

EPOCH 5 ...
Training Accuracy = 0.058
Validation Accuracy = 0.058

Training on RGB images processed as Normalized Data

EPOCH 1 ...
Training Accuracy = 0.058
Validation Accuracy = 0.054

EPOCH 2 ...
Training Accuracy = 0.058
Validation Accuracy = 0.054

EPOCH 3 ...
Training Accuracy = 0.058
Validation Accuracy = 0.054

EPOCH 4 ...
Training Accuracy = 0.058
Validation Accuracy = 0.054

EPOCH 5 ...
Training Accuracy = 0.058
Validation Accuracy = 0.054

但是，如果我重新启动内核并使用任何数据类型(任何迭代)，它就会起作用。我发现我必须清除图表或为多种数据类型运行多个 session ，但我还不清楚如何做到这一点。我尝试使用 tf.reset_default_graph() ，但似乎没有任何效果。有人能指出我正确的方向吗？

谢谢

Answer 1

您可能希望在将数据输入网络之前尝试使用归一化为零均值和单位方差的数据，例如将图像缩放至-1..1范围;也就是说，0..1范围大多听起来也很正常。根据网络中使用的激活，值范围可能会产生很大的差异:例如，ReLU 在输入低于零时消失，当值低于 -4 时 sigmoid 开始饱和。或以上+4如果没有值低于 0，则 tanh 激活会错过其值范围的一半- 如果值范围太大，梯度也可能爆炸，从而完全阻止训练。来自 this paper ，作者似乎减去了(批量)图像平均值而不是值范围平均值。

您也可以尝试使用较小的学习率(尽管就我个人而言，我通常开始对 Adam 的 0.0001 进行实验)。

至于问题的多个 session 部分:它当前在代码中实现的方式，基本上使默认图表变得困惑。通过调用

for key in finalInputdata.keys():
    for procTypes in range(0,(len(finalInputdata[key]))):
        if np.shape(finalInputdata[key][procTypes][0]) != ():

    # ...

    x = tf.placeholder(tf.float32, (None, 32, 32,np.shape(X_train)[-1]))
    y = tf.placeholder(tf.int32, (None))
    one_hot_y = tf.one_hot(y,43)

    # Tensor Operations
    logits = LeNet(x,np.shape(X_train)[-1])

    # ... etc ...

您正在创建len(finalInputdata) * N LeNet 的不同实例，都在默认图中。当变量在网络内部重用时，这可能是一个问题。

如果您确实想重置默认图表以尝试不同的超参数，请尝试

for key in finalInputdata.keys():
    for procTypes in range(0,(len(finalInputdata[key]))):

        tf.reset_default_graph()
        # define the graph

        sess = tf.InteractiveSession()
        # train

但最好像这样显式创建图形和 session :

for key in finalInputdata.keys():
    for procTypes in range(0,(len(finalInputdata[key]))):

        with tf.Graph().as_default() as graph:
            # define the graph

        with tf.Session(graph=graph) as sess:
            # train

而不是打电话sess = tf.get_default_session()然后你可以直接使用sess引用。

我还发现 Jupyter 内核和支持 GPU 的 TensorFlow 在网络上迭代时不能很好地协同工作，有时会遇到内存不足错误或彻底崩溃浏览器选项卡。