python - 如何重置 Keras 指标？

Question

为了做一些参数调整，我喜欢用 Keras 循环一些训练函数。但是，我意识到在使用 tensorflow.keras.metrics.AUC() 时作为度量，对于每个训练循环，都会将一个整数添加到 auc 度量名称(例如 auc_1、auc_2、...)。所以实际上，即使从训练函数中出来，keras 指标也会以某种方式存储。

这首先导致回调不再识别指标，并且还让我怀疑是否有其他东西存储，如模型权重。

如何重置指标以及是否有其他由 keras 存储的内容需要重置才能重新启动以进行培训？

您可以在下面找到一个最小的工作示例:

编辑:这个例子似乎只适用于 tensorflow 2.2

import numpy as np
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.metrics import AUC


def dummy_network(input_shape):
    model = keras.Sequential()
    model.add(keras.layers.Dense(10,
                                 input_shape=input_shape,
                                 activation=tf.nn.relu,
                                 kernel_initializer='he_normal',
                                 kernel_regularizer=keras.regularizers.l2(l=1e-3)))

    model.add(keras.layers.Flatten())
    model.add(keras.layers.Dense(11, activation='sigmoid'))

    model.compile(optimizer='adagrad',
                  loss='binary_crossentropy',
                  metrics=[AUC()])
    return model


def train():
    CB_lr = tf.keras.callbacks.ReduceLROnPlateau(
        monitor="val_auc",
        patience=3,
        verbose=1,
        mode="max",
        min_delta=0.0001,
        min_lr=1e-6)

    CB_es = tf.keras.callbacks.EarlyStopping(
        monitor="val_auc",
        min_delta=0.00001,
        verbose=1,
        patience=10,
        mode="max",
        restore_best_weights=True)
    callbacks = [CB_lr, CB_es]
    y = [np.ones((11, 1)) for _ in range(1000)]
    x = [np.ones((37, 12, 1)) for _ in range(1000)]
    dummy_dataset = tf.data.Dataset.from_tensor_slices((x, y)).batch(batch_size=100).repeat()
    val_dataset = tf.data.Dataset.from_tensor_slices((x, y)).batch(batch_size=100).repeat()
    model = dummy_network(input_shape=((37, 12, 1)))
    model.fit(dummy_dataset, validation_data=val_dataset, epochs=2,
              steps_per_epoch=len(x) // 100,
              validation_steps=len(x) // 100, callbacks=callbacks)


for i in range(3):
    print(f'\n\n **** Loop {i} **** \n\n')
    train()

输出是:

 **** Loop 0 **** 


2020-06-16 14:37:46.621264: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f991e541f10 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-16 14:37:46.621296: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
Epoch 1/2
10/10 [==============================] - 0s 44ms/step - loss: 0.1295 - auc: 0.0000e+00 - val_loss: 0.0310 - val_auc: 0.0000e+00 - lr: 0.0010
Epoch 2/2
10/10 [==============================] - 0s 10ms/step - loss: 0.0262 - auc: 0.0000e+00 - val_loss: 0.0223 - val_auc: 0.0000e+00 - lr: 0.0010


 **** Loop 1 **** 


Epoch 1/2
10/10 [==============================] - ETA: 0s - loss: 0.4751 - auc_1: 0.0000e+00WARNING:tensorflow:Reduce LR on plateau conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_1,val_loss,val_auc_1,lr
WARNING:tensorflow:Early stopping conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_1,val_loss,val_auc_1,lr
10/10 [==============================] - 0s 36ms/step - loss: 0.4751 - auc_1: 0.0000e+00 - val_loss: 0.3137 - val_auc_1: 0.0000e+00 - lr: 0.0010
Epoch 2/2
10/10 [==============================] - ETA: 0s - loss: 0.2617 - auc_1: 0.0000e+00WARNING:tensorflow:Reduce LR on plateau conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_1,val_loss,val_auc_1,lr
WARNING:tensorflow:Early stopping conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_1,val_loss,val_auc_1,lr
10/10 [==============================] - 0s 10ms/step - loss: 0.2617 - auc_1: 0.0000e+00 - val_loss: 0.2137 - val_auc_1: 0.0000e+00 - lr: 0.0010


 **** Loop 2 **** 


Epoch 1/2
10/10 [==============================] - ETA: 0s - loss: 0.1948 - auc_2: 0.0000e+00WARNING:tensorflow:Reduce LR on plateau conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_2,val_loss,val_auc_2,lr
WARNING:tensorflow:Early stopping conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_2,val_loss,val_auc_2,lr
10/10 [==============================] - 0s 34ms/step - loss: 0.1948 - auc_2: 0.0000e+00 - val_loss: 0.0517 - val_auc_2: 0.0000e+00 - lr: 0.0010
Epoch 2/2
10/10 [==============================] - ETA: 0s - loss: 0.0445 - auc_2: 0.0000e+00WARNING:tensorflow:Reduce LR on plateau conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_2,val_loss,val_auc_2,lr
WARNING:tensorflow:Early stopping conditioned on metric `val_auc` which is not available. Available metrics are: loss,auc_2,val_loss,val_auc_2,lr
10/10 [==============================] - 0s 10ms/step - loss: 0.0445 - auc_2: 0.0000e+00 - val_loss: 0.0389 - val_auc_2: 0.0000e+00 - lr: 0.0010

Answer 1

您的可重现示例在几个地方失败了，所以我只更改了一些内容(我使用的是 TF 2.1)。让它运行后，我可以通过指定 metrics=[AUC(name='auc')] 去掉额外的度量名称。 .这是完整的(固定的)可重现示例:

import numpy as np
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.metrics import AUC


def dummy_network(input_shape):
    model = keras.Sequential()
    model.add(keras.layers.Dense(10,
                                 input_shape=input_shape,
                                 activation=tf.nn.relu,
                                 kernel_initializer='he_normal',
                                 kernel_regularizer=keras.regularizers.l2(l=1e-3)))

    model.add(keras.layers.Flatten())
    model.add(keras.layers.Dense(11, activation='softmax'))

    model.compile(optimizer='adagrad',
                  loss='binary_crossentropy',
                  metrics=[AUC(name='auc')])
    return model


def train():
    CB_lr = tf.keras.callbacks.ReduceLROnPlateau(
        monitor="val_auc",
        patience=3,
        verbose=1,
        mode="max",
        min_delta=0.0001,
        min_lr=1e-6)

    CB_es = tf.keras.callbacks.EarlyStopping(
        monitor="val_auc",
        min_delta=0.00001,
        verbose=1,
        patience=10,
        mode="max",
        restore_best_weights=True)
    callbacks = [CB_lr, CB_es]
    y = tf.keras.utils.to_categorical([np.random.randint(0, 11) for _ in range(1000)])
    x = [np.ones((37, 12, 1)) for _ in range(1000)]
    dummy_dataset = tf.data.Dataset.from_tensor_slices((x, y)).batch(batch_size=100).repeat()
    val_dataset = tf.data.Dataset.from_tensor_slices((x, y)).batch(batch_size=100).repeat()
    model = dummy_network(input_shape=((37, 12, 1)))
    model.fit(dummy_dataset, validation_data=val_dataset, epochs=2,
              steps_per_epoch=len(x) // 100,
              validation_steps=len(x) // 100, callbacks=callbacks)


for i in range(3):
    print(f'\n\n **** Loop {i} **** \n\n')
    train()

Train for 10 steps, validate for 10 steps
Epoch 1/2
 1/10 [==>...........................] - ETA: 6s - loss: 0.3426 - auc: 0.4530
 7/10 [====================>.........] - ETA: 0s - loss: 0.3318 - auc: 0.4895
10/10 [==============================] - 1s 117ms/step - loss: 0.3301 - 
                                         auc: 0.4893 - val_loss: 0.3222 - val_auc: 0.5085

发生这种情况是因为每个循环，您都通过执行以下操作创建了一个没有指定名称的新指标: metrics=[AUC()] .在循环的第一次迭代中，TF 自动在命名空间中创建了一个名为 auc 的变量。 ，但在循环的第二次迭代中，名称 'auc'已经被占用，所以 TF 将其命名为 auc_1因为您没有指定名称。但是，您的回调被设置为基于 auc ，这是该模型没有的度量(它是上一个循环中模型的度量)。所以，你要么做 name='auc'并覆盖先前的指标名称，或在循环之外定义它，如下所示:

import numpy as np
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.metrics import AUC

auc = AUC()

def dummy_network(input_shape):
    model = keras.Sequential()
    model.add(keras.layers.Dense(10,
                                 input_shape=input_shape,
                                 activation=tf.nn.relu,
                                 kernel_initializer='he_normal',
                                 kernel_regularizer=keras.regularizers.l2(l=1e-3)))

    model.add(keras.layers.Flatten())
    model.add(keras.layers.Dense(11, activation='softmax'))
    model.compile(optimizer='adagrad',
                  loss='binary_crossentropy',
                  metrics=[auc])
    return model

不用担心 keras重置指标。它负责处理 fit() 中的所有内容方法。如果你想要更多的灵活性和/或自己做，我建议使用自定义训练循环，并自己重置:

auc = tf.keras.metrics.AUC()

auc.update_state(np.random.randint(0, 2, 10), np.random.randint(0, 2, 10)) 

print(auc.result())

auc.reset_states()

print(auc.result())

Out[6]: <tf.Tensor: shape=(), dtype=float32, numpy=0.875>  # state updated

Out[8]: <tf.Tensor: shape=(), dtype=float32, numpy=0.0>  # state reset