python - TensorFlow, tf.reshape 导致 "Gradients do not exist for variables"

Question

我正在使用 Tensorflow/Keras(TF 版本 2.1，Windows 上的 Python 3.7)编写一个完全连接的层，但我发现如果我在乘以之前 reshape 权重张量，那么 Tensorflow 似乎不是即使我只是 reshape 为自己的形状，也能够计算梯度。考虑以下层代码:

import tensorflow as tf
import numpy as np

class FCLayer(tf.keras.layers.Layer):
    def __init__(self,output_size,cause_error = False):
        super(FCLayer,self).__init__()
        self.output_size = output_size
        self.cause_error = cause_error

    def build(self,input_shape): 
        self.input_size = input_shape[1]        
        weights = self.add_weight(shape=(self.input_size,
                                         self.output_size),
                                 initializer='random_normal',
                                 trainable=True)

        if self.cause_error:
            self.weights2 = tf.reshape( weights,
                                        shape = (self.input_size,
                                                 self.output_size))
        else:
            self.weights2 = weights

    def call(self, inputs):
        return tf.matmul(inputs, self.weights2)    

如果这与 cause_error = True 一起使用，那么在 mnist 上训练 4 个 epoch 时会得到以下输出(具体训练代码如下):

Train on 60000 samples, validate on 10000 samples
Epoch 1/4
WARNING:tensorflow:Gradients do not exist for variables ['sequential/dummy_layer/Variable:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['sequential/dummy_layer/Variable:0'] when minimizing the loss.
60000/60000 [==============================] - 1s 20us/sample - loss: 2.4131 - accuracy: 0.0722 - val_loss: 2.3963 - val_accuracy: 0.0834
Epoch 2/4
60000/60000 [==============================] - 1s 12us/sample - loss: 2.4122 - accuracy: 0.0722 - val_loss: 2.3953 - val_accuracy: 0.0836
Epoch 3/4
60000/60000 [==============================] - 1s 12us/sample - loss: 2.4112 - accuracy: 0.0724 - val_loss: 2.3944 - val_accuracy: 0.0838
Epoch 4/4
60000/60000 [==============================] - 1s 13us/sample - loss: 2.4102 - accuracy: 0.0725 - val_loss: 2.3933 - val_accuracy: 0.0839

这只是一个警告，但很明显模型并没有真正改进，显然它需要那些梯度。

如果我设置了 cause_error=False 我会得到预期的输出(没有警告，适度的改进):

Train on 60000 samples, validate on 10000 samples
Epoch 1/4
60000/60000 [==============================] - 1s 16us/sample - loss: 2.3671 - accuracy: 0.1527 - val_loss: 2.3445 - val_accuracy: 0.1508
Epoch 2/4
60000/60000 [==============================] - 1s 12us/sample - loss: 2.3293 - accuracy: 0.1596 - val_loss: 2.3072 - val_accuracy: 0.1610
Epoch 3/4
60000/60000 [==============================] - 1s 13us/sample - loss: 2.2939 - accuracy: 0.1683 - val_loss: 2.2722 - val_accuracy: 0.1720
Epoch 4/4
60000/60000 [==============================] - 1s 13us/sample - loss: 2.2609 - accuracy: 0.1784 - val_loss: 2.2397 - val_accuracy: 0.1847

我怀疑我需要以某种方式告诉 Tensorflow 来跟踪渐变，但不太确定如何。当我使用 tf.matmul 时，它似乎会自动完成，而且我很确定这种代码曾经在 TF 1 中工作过。

我用来执行的具体代码是(改编自mnist教程):

batch_size = 128
num_classes = 10
epochs = 4

# input image dimensions
img_rows, img_cols = 28, 28

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()


x_train = x_train.reshape(x_train.shape[0], img_rows* img_cols)
x_test = x_test.reshape(x_test.shape[0], img_rows*img_cols)
input_shape = (img_rows * img_cols)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
y_train = tf.keras.utils.to_categorical(y_train, num_classes)
y_test = tf.keras.utils.to_categorical(y_test, num_classes)

model = tf.keras.models.Sequential()

dummy_layer = FCLayer(10, cause_error = True)
model.add( dummy_layer )
model.add( tf.keras.layers.Dense(10, activation='softmax') )

model.compile(loss=tf.keras.losses.categorical_crossentropy,
              optimizer=tf.keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

Answer 1

该问题与 TF 2.0 急切执行有关——任何诸如 tf.reshape 之类的操作都会在遇到它们的那一刻运行。 build 仅对给定模型调用一次。现在，发生的事情是您正在创建一个张量 weights2，它是 tf.Variable weights 的 reshape 版本，但它是 not 本身是 tf.Variable (操作通常返回张量，而不是变量)。因为这发生在急切执行中，所以不会保留任何“记录”，并且 weights2 与 weights 没有任何联系。因此，在模型调用中使用时，weights 无法更新。这在 else 的情况下不会发生，因为这里 weights2 只是另一个名称，指代实际的 tf.Variable weights.

两种解决方法:

1. 在 build 中使用 assign 进行就地 reshape (注意，我使用 self.w 因为 self .weights 是 Keras 层的保留名称):

   def build(self,input_shape): 
       self.input_size = input_shape[1]        
       self.w = self.add_weight(shape=(self.input_size,
                                             self.output_size),
                                      initializer='random_normal',
                                      trainable=True)
   
       if self.cause_error:
           self.w.assign(tf.reshape(self.w,
                                      shape = (self.input_size,
                                               self.output_size)))
   

这不会导致错误/警告，但它可能不是您想要的，因为您正在修改丢失的原始 weights。我想您希望在每次调用时都使用 weights 的修改版本。在这种情况下，请在 call 方法中执行:

class FCLayer(tf.keras.layers.Layer):
    def __init__(self,output_size,cause_error = False):
        super(FCLayer,self).__init__()
        self.output_size = output_size
        self.cause_error = cause_error

    def build(self,input_shape): 
        self.input_size = input_shape[1]        
        self.w = self.add_weight(shape=(self.input_size,
                                          self.output_size),
                                   initializer='random_normal',
                                   trainable=True)
    def call(self, inputs):
        weights2 = tf.reshape(self.w, (self.input_size, self.output_size)
        return tf.matmul(inputs, weights2)

这是可行的，因为现在 reshape 操作是模型调用图的一部分，即我们可以回溯 weights2 实际上来自 weights，并且渐变可以流动。