python - tensorflow 推理时的批量归一化

Question

我已经加载了经过训练的检查点文件以进行推理。我已经从模型中提取了贝塔值、移动均值和移动方差以及所有权重。在批量标准化中，当我手动计算 batch_normalization 的输出时，我得到了错误的结果。[已更新]

在这里，我分享我的代码，它加载检查点，打印批量归一化的输入，打印 beta，移动均值和移动方差，并在控制台上打印批量归一化的输出。

import tensorflow as tf
import cv2
import numpy as np
import time
import os

def main():
    with tf.Session() as sess:        

        #[INFO] code for loading checkpoint
        #---------------------------------------------------------------------
        saver = tf.train.import_meta_graph("./bag-model-34000.meta")
        saver.restore(sess, tf.train.latest_checkpoint("./"))
        graph = tf.get_default_graph()
        input_place = graph.get_tensor_by_name('input/image_input:0')
        op = graph.get_tensor_by_name('output/image_output:0')
        #----------------------------------------------------------------------

        #[INFO] generating input data which is equal to input tensor shape
        #----------------------------------------------------------------------
        input_data = np.random.randint(255, size=(1,320,240, 3)).astype(float)
        #----------------------------------------------------------------------

        #[INFO] code to get all tensors_name
        #----------------------------------------------------------------------
        operations = sess.graph.get_operations()
        ind = 0;
        tens_name = []  # store all tensor name in list
        for operation in operations:
            #print(ind,"> ", operation.name, "=> \n", operation.values())

            if (operation.values()): 
                name_of_tensor = str(operation.values()).split()[1][1:-1]

            tens_name.append(name_of_tensor)
            ind = ind + 1
        #------------------------------------------------------------------------

        #[INFO] printing Input to batch normalization, beta, moving mean and moving variance
        # so I can calculate manually batch normalization output
        #------------------------------------------------------------------------   
        tensor_number = 0
        for tname in tens_name:         # looping through each tensor name

            if tensor_number <= 812:      # I am interested in first 812 tensors
                tensor = graph.get_tensor_by_name(tname)
                tensor_values = sess.run(tensor, feed_dict={input_place: input_data})
                print("tensor: ", tensor_number, ": ", tname, ": \n\t\t", tensor_values.shape)


                # [INFO] 28'th tensor its name is "input/conv1/conv1_1/separable_conv2d:0"
                # the output of this tensor is input to the batch normalization
                if tensor_number == 28:
                    # here I am printing this tensor output
                    print(tensor_values)            # [[[[-0.03182551  0.00226904  0.00440771 ... 
                    print(tensor_values.shape)      # (1, 320, 240, 32)


                # [INFO] 31'th tensor its name is "conv1/conv1_1/BatchNorm/beta:0"
                # the output of this tensor is all beta
                if tensor_number == 31:
                    # here I am printing this beta's
                    print(tensor_values)            # [ 0.04061257 -0.16322449 -0.10942575 ...
                    print(tensor_values.shape)      # (32,)


                # [INFO] 35'th tensor its name is "conv1/conv1_1/BatchNorm/moving_mean:0"
                # the output of this tensor is all moving mean
                if tensor_number == 35:
                    # here I am printing this moving means
                    print(tensor_values)            # [-0.0013569   0.00618145  0.00248459 ...
                    print(tensor_values.shape)      # (32,)


                # [INFO] 39'th tensor its name is "conv1/conv1_1/BatchNorm/moving_variance:0"
                # the output of this tensor is all moving_variance
                if tensor_number == 39:
                    # here I am printing this moving variance
                    print(tensor_values)            # [4.48082483e-06 1.21615967e-05 5.37582537e-06 ...
                    print(tensor_values.shape)      # (32,)


                # [INFO] 44'th tensor its name is "input/conv1/conv1_1/BatchNorm/FusedBatchNorm:0"
                # here perform batch normalization and here I am printing the output of this tensor
                if tensor_number == 44:
                    # here I am printing the output of this tensor
                    print(tensor_values)            # [[[[-8.45019519e-02  1.23237416e-01 -4.60943699e-01 ...
                    print(tensor_values.shape)      # (1, 320, 240, 32)

            tensor_number = tensor_number + 1
        #---------------------------------------------------------------------------------------------

if __name__ == "__main__":
    main()

因此，从控制台运行上述代码后，我得到了批量归一化的输入，这是“input/conv1/conv1_1/separable_conv2d:0”这个张量的输出。

I am taking the first value from that output as x,
so, input x = -0.03182551

and beta, moving mean and moving variance is also printed on console. 
and I am take the first value from each array.
                beta = 0.04061257
                moving mean = -0.0013569
                moving variance = 4.48082483e-06
                epsilon = 0.001  ... It is default value

and gamma is ignored. because I set training time as scale = false so gamma is ignored.

When I am calculate the output of batch normalization at inference time for given input x
x_hat = (x - moving_mean) / square_root_of(moving variance + epsilon)
      = (-0.03182551 − (-0.0013569)) / √(0.00000448082483 + 0.001)
      = −0.961350647
so x_hat is −0.961350647

y = gamma * x_hat + beta
gamma is ignored so equation becomes y = x_hat + beta
                                       = −0.961350647 + 0.04061257
                                     y = −0.920738077

So If I calculated manually y at inference time it gives as y = −0.920738077
but in program it showing y = -8.45019519e-02
It is output of "input/conv1/conv1_1/BatchNorm/FusedBatchNorm:0" tensor.

It is very very different from what I am calculated. Is my equation is wrong? So which modifications 
I have to make to above x_hat and y equation so I can get this value.

所以，我很困惑为什么我的计算结果与结果值非常不同？

我还使用 tf.compat.v1.global_variables() 检查了 beta、移动均值和移动方差。所有值都与控制台上打印的 beta、移动均值和移动方差值相匹配。

那么为什么我在等式x_hat和y中手动替换这些值后得到错误的结果？

我还在这里提供我的控制台输出，从tensor_number 28到44...

tensor:  28 :  input/conv1/conv1_1/separable_conv2d:0 : 
                 (1, 320, 240, 32)
[[[[-0.03182551  0.00226904  0.00440771 ... -0.01204819  0.02620635

tensor:  29 :  input/conv1/conv1_1/BatchNorm/Const:0 : 
                 (32,)
tensor:  30 :  conv1/conv1_1/BatchNorm/beta/Initializer/zeros:0 : 
                 (32,)

tensor:  31 :  conv1/conv1_1/BatchNorm/beta:0 : 
                 (32,)
[ 0.04061257 -0.16322449 -0.10942575  0.05056419 -0.13785222  0.4060304

tensor:  32 :  conv1/conv1_1/BatchNorm/beta/Assign:0 : 
                 (32,)
tensor:  33 :  conv1/conv1_1/BatchNorm/beta/read:0 : 
                 (32,)
tensor:  34 :  conv1/conv1_1/BatchNorm/moving_mean/Initializer/zeros:0 : 
                 (32,)

tensor:  35 :  conv1/conv1_1/BatchNorm/moving_mean:0 : 
                 (32,)
[-0.0013569   0.00618145  0.00248459  0.00340403  0.00600711  0.00291052

tensor:  36 :  conv1/conv1_1/BatchNorm/moving_mean/Assign:0 : 
                 (32,)
tensor:  37 :  conv1/conv1_1/BatchNorm/moving_mean/read:0 : 
                 (32,)
tensor:  38 :  conv1/conv1_1/BatchNorm/moving_variance/Initializer/ones:0 : 
                 (32,)

tensor:  39 :  conv1/conv1_1/BatchNorm/moving_variance:0 : 
                 (32,)
[4.48082483e-06 1.21615967e-05 5.37582537e-06 1.40261754e-05

tensor:  40 :  conv1/conv1_1/BatchNorm/moving_variance/Assign:0 : 
                 (32,)
tensor:  41 :  conv1/conv1_1/BatchNorm/moving_variance/read:0 : 
                 (32,)
tensor:  42 :  input/conv1/conv1_1/BatchNorm/Const_1:0 : 
                 (0,)
tensor:  43 :  input/conv1/conv1_1/BatchNorm/Const_2:0 : 
                 (0,)

tensor:  44 :  input/conv1/conv1_1/BatchNorm/FusedBatchNorm:0 : 
                 (1, 320, 240, 32)
[[[[-8.45019519e-02  1.23237416e-01 -4.60943699e-01 ...  3.77691090e-01

Answer 1

我解决了这个问题，对于批量归一化操作，它认为它正在训练中。

因此，它使用批量均值、批量方差和 beta 为 0，而不是提供移动均值、移动方差和 beta。

因此，我计算了批处理均值、批处理方差，并将这些值代入方程中，现在它给出了正确的输出。

那么如何强制他使用移动均值和移动方差以及提供的贝塔值呢？我尝试通过将训练设置为 false 来进行此更改。但它不起作用。

for tname in tens_name:         # looping through each tensor name

            if tensor_number <= 812:      # I am interested in first 812 tensors
                training = tf.placeholder(tf.bool, name = 'training')
                is_training = tf.placeholder(tf.bool, name = 'is_training')
                tensor = graph.get_tensor_by_name(tname)
                tensor_values = sess.run(tensor, feed_dict={is_training: False, training: False, input_place: input_data})

在实际代码中 is_training 为 true

def load_cnn(self,keep_prob = 0.5, num_filt = 32, num_layers = 2,is_training=True):
        self.reuse=False
        with tf.name_scope('input'):
            self.image_input=tf.placeholder(tf.float32,shape=[None,None,None,3],name='image_input')
            net=self.image_input

            with slim.arg_scope([slim.separable_conv2d],
            depth_multiplier=1,
            normalizer_fn=slim.batch_norm,
            normalizer_params={'is_training':is_training},
            activation_fn=tf.nn.relu,weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),
            weights_regularizer=slim.l2_regularizer(0.0005)):

                # Down Scaling
                # Block 1
                net=slim.repeat(net, 2, slim.separable_conv2d, num_filt, [3, 3], scope = 'conv1')
                print('en_conv1',net.shape,net.name) # 320x240x3 -> 316x236x32
                self.cnn_layer1=net
                #Down Sampling
                net=slim.max_pool2d(net,[2,2],scope='pool1') 
                print('en_maxpool1',net.shape,net.name) # 316x236x32 -> 158x118x32