python - 未能通过 GradientDescentOptimizer 找到零权重

Question

我尝试构建一个回归模型来通过 TensorFlow 训练我的数据集。当为W1*x^2 + W2*x + b时，显示nan；当为W2*x+b时，可以输出数字。为什么找不到W1=0？我的模型构建逻辑有什么问题吗？

import tensorflow as tf
import csv
import re
import datetime
import numpy
import matplotlib.pyplot as plt

# Parameters
learning_rate = 0.01
training_epochs = 2000

# Training Data
data_X = [  0.,   2.,   5.,   6.,   7.,   8.,   9.,  12.,  13.,  14.]
data_Y = [ 2568.300049,  2540.100098,  2552.399902,  2583.899902,  2607.100098,
  2603.300049,  2561.699951,  2614.899902,  2590.800049,  2578.199951]
train_X = numpy.asarray(data_X)
train_Y = numpy.asarray(data_Y)
n_samples = train_X.shape[0]

# Model parameters
rng = numpy.random
W1 = tf.Variable([rng.randn()], dtype=tf.float32, name="weight1")
# OK when W1 = tf.constant(0.)
W2 = tf.Variable([rng.randn()], dtype=tf.float32, name="weight2")
b = tf.Variable([rng.randn()], dtype=tf.float32, name="bias")
# Model input and output
x = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)
linear_model = W1*tf.square(x) + W2*x + b

# loss
loss = tf.reduce_sum(tf.square(linear_model - y))/(2*n_samples)
# optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
train = optimizer.minimize(loss)

# training loop
init = tf.global_variables_initializer()
# Start training
with tf.Session() as sess:
    sess.run(init) # reset values to wrong
    for i in range(training_epochs):
        sess.run(train, {x: train_X, y: train_Y})
    # evaluate training accuracy
    curr_W1, curr_W2, curr_b, curr_loss = sess.run([W1, W2, b, loss], {x: train_X, y: train_Y})
    print("W1: %s W2: %s b: %s loss: %s"%(curr_W1, curr_W2, curr_b, curr_loss))

    # Graphic display
    plt.plot(train_X, train_Y, 'ro', label='Original data')
    plt.plot(train_X, sess.run(W1) * numpy.square(train_X) + sess.run(W2) * train_X + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

Answer 1

约尔·泽尔德斯是正确的。然而，这么简单的问题不应该需要 2000000 个训练步骤。这样做的原因首先是因为二次模型不能很好地匹配明显的线性数据，其次是因为最佳解决方案需要巨大的偏差值，这需要很长时间才能学习，因为 更改的影响b 与 W1 更改的影响相比显得相形见绌。

解决这个问题的一个好方法是重新缩放数据。如果您包含这些行

train_X = (train_X - numpy.mean(train_X)) / (numpy.amax(train_X) - numpy.amin(train_X))
train_Y = (train_Y - numpy.mean(train_Y)) / (numpy.amax(train_Y) - numpy.amin(train_Y))`

那么你的数据的平均值为 0，范围为 1，并且使用任一模型进行训练都会容易得多。请注意，如果您想在新数据点上测试经过训练的模型，则需要将它们缩放相同的量。