tensorflow - 加载 tensorflow 模型后运行forward prop函数

Question

加载保存的 Tensorflow 模型后，我无法运行前向传播函数。我能够成功提取权重，但是当我尝试将新输入传递给前向 Prop 函数时，它会抛出“尝试使用未初始化的值”错误。

我的占位符如下:

x = tf.placeholder('int64', [None, 4], name='input')  # Number of examples x features
y = tf.placeholder('int64', [None, 1], name='output')  # Number of examples x output

正向 Prop 功能:

def forwardProp(x, y):

    embedding_mat = tf.get_variable("EM", shape=[total_vocab, e_features], initializer=tf.random_normal_initializer(seed=1))

    # m x words x total_vocab * total_vocab x e_features = m x words x e_features
    # embed_x = tf.tensordot(x, tf.transpose(embedding_mat), axes=[[2], [0]])
    # embed_y = tf.tensordot(y, tf.transpose(embedding_mat), axes=[[2], [0]])

    embed_x = tf.gather(embedding_mat, x)  # m x words x e_features
    embed_y = tf.gather(embedding_mat, y)  # m x words x e_features

    #print("Shape of embed x", embed_x.get_shape())

    W1 = tf.get_variable("W1", shape=[n1, e_features], initializer=tf.random_normal_initializer(seed=1))
    B1 = tf.get_variable("b1", shape=[1, 4, n1], initializer=tf.zeros_initializer())

    # m x words x e_features *  e_features x n1 = m x words x n1
    Z1 = tf.add(tf.tensordot(embed_x, tf.transpose(W1), axes=[[2], [0]]), B1, )
    A1 = tf.nn.tanh(Z1)

    W2 = tf.get_variable("W2", shape=[n2, n1], initializer=tf.random_normal_initializer(seed=1))
    B2 = tf.get_variable("B2", shape=[1, 4, n2], initializer=tf.zeros_initializer())

    # m x words x n1 *  n1 x n2 = m x words x n2
    Z2 = tf.add(tf.tensordot(A1, tf.transpose(W2), axes=[[2], [0]]), B2)
    A2 = tf.nn.tanh(Z2)

    W3 = tf.get_variable("W3", shape=[n3, n2], initializer=tf.random_normal_initializer(seed=1))
    B3 = tf.get_variable("B3", shape=[1, 4, n3], initializer=tf.zeros_initializer())

    # m x words x n2  * n2 x n3 = m x words x n3
    Z3 = tf.add(tf.tensordot(A2, tf.transpose(W3), axes=[[2], [0]]), B3)
    A3 = tf.nn.tanh(Z3)

    # Convert m x words x n3 to m x n3

    x_final = tf.reduce_mean(A3, axis=1)
    y_final = tf.reduce_mean(embed_y, axis=1)

    return x_final, y_final

返回 Prop 功能:

def backProp(X_index, Y_index):
    x_final, y_final = forwardProp(x, y)
    cost = tf.nn.l2_loss(x_final - y_final)
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    init = tf.global_variables_initializer()
    saver = tf.train.Saver()
    total_batches = math.floor(m/batch_size)


    with tf.Session() as sess:
        sess.run(init)

        for epoch in range(epochs):
            batch_start = 0

            for i in range(int(m/batch_size)):

                x_hot = X_index[batch_start: batch_start + batch_size]
                y_hot = Y_index[batch_start: batch_start + batch_size]
                batch_start += batch_size

                _, temp_cost = sess.run([optimizer, cost], feed_dict={x: x_hot, y: y_hot})

                print("Cost at minibatch:  ", i , " and epoch ", epoch, " is ", temp_cost)

            if m % batch_size != 0:
                x_hot = X_index[batch_start: batch_start+m - (batch_size*total_batches)]
                y_hot = Y_index[batch_start: batch_start+m - (batch_size*total_batches)]
                _, temp_cost = sess.run([optimizer, cost], feed_dict={x: x_hot, y: y_hot})
                print("Cost at minibatch: (beyond floor)  and epoch ", epoch, " is ", temp_cost)


        # Saving the model
        save_path = saver.save(sess, "./model_neural_embeddingV1.ckpt")
        print("Model saved!")

通过调用预测函数重新加载模型:

def predict_search():

    # Initialize variables
    total_features = 4
    extra = len(word_to_indice)
    query = input('Enter your query')
    words = word_tokenize(query)
    # For now, it will throw an error if a word not present in dictionary is present
    features = [word_to_indice[w.lower()] for w in words]
    len_features = len(features)
    X_query = []
    Y_query = [[0]]  # Dummy variable, we don't care about the Y query while doing prediction
    if len_features < total_features:
        features += [extra] * (total_features - len_features)
    elif len_features > total_features:
        features = features[:total_features]

    X_query.append(features)
    X_query = np.array(X_query)
    print(X_query)
    Y_query = np.array(Y_query)

    # Load the model

    init_global = tf.global_variables_initializer()
    init_local = tf.local_variables_initializer()

    #X_final, Y_final = forwardProp(x, y)

    with tf.Session() as sess:
        sess.run(init_global)
        sess.run(init_local)
        saver = tf.train.import_meta_graph('./model_neural_embeddingV1.ckpt.meta')
        saver.restore(sess, './model_neural_embeddingV1.ckpt')
        print("Model loaded")
        print("Loaded variables are: ")
        print(tf.trainable_variables())
        print(sess.graph.get_operations())
        embedMat = sess.run('EM:0')  # Get the word embedding matrix
        W1 = sess.run('W1:0')
        b1 = sess.run('b1:0')
        W2 = sess.run('W2:0')
        b2 = sess.run('B2:0')
        print(b2)
        W3 = sess.run('W3:0')
        b3 = sess.run('B3:0')

        **#This part is not working, calling forward prop gives an 'attempting to use uninitialized value' error.** 
        X_final = sess.run(forwardProp(x, y), feed_dict={x: X_query, y: Y_query})

        print(X_final)

Answer 1

从元图加载后，您不小心使用 forwardProp 函数创建了一堆图变量，从而在无意中有效地复制了变量。

您应该重构代码，以遵循在创建 session 之前创建图形变量的最佳实践。

例如，在名为 build_graph 的函数中创建所有变量。您可以在创建 session 之前调用 build_graph，但绝对不能在创建 session 之后调用。这将避免这样的困惑。

您几乎应该始终避免从 sess.run 调用函数，例如您正在执行的操作:

X_final = sess.run(forwardProp(x, y), feed_dict={x: X_query, y: Y_query})

您就是在以这种方式寻求错误。

请注意在 forwardProp(x, y) 中发生的情况，您正在创建 tensorflow 构造、所有权重和偏差。

但请注意，您是在这两行代码中创建的:

saver = tf.train.import_meta_graph('./model_neural_embeddingV1.ckpt.meta')
saver.restore(sess, './model_neural_embeddingV1.ckpt')

另一个选项(可能是您想要做的)是不使用import_meta_graph。您可以创建所有 TensorFlow OP 和变量，然后运行 ​​saver.restore 来恢复检查点，这会将检查点数据映射到您已创建的变量中。

请注意，这里的 tensorflow 实际上有 2 个选项，这有点令人困惑。您最终完成了这两件事(导入包含所有操作和变量的图表)，并重新创建图表。你必须选择一个。

我通常选择第一个选项，不使用 import_meta_graph，只需通过调用 build_graph 函数以编程方式重新创建图表。然后调用 saver.restore 引入检查点。当然，您将重复使用 build_graph 函数进行训练和推理时间，因此您最终会得到两次都是相同的图表。