python - 提早停止 TensorFlow 2.0

Question

我正在使用 Python 3.7.5 和 TensorFlow 2.0 运行代码进行 MNIST 分类。我正在使用 TensorFlow 2.0 中的 EarlyStopping，我的回调是:

callbacks = [
             tf.keras.callbacks.EarlyStopping(
                 monitor='val_loss', patience = 3,
                 min_delta=0.001
             )
]

根据EarlyStopping - TensorFlow 2.0页面中，min_delta参数定义如下:

min_delta:监控数量的最小变化才有资格作为改进，即绝对变化小于 min_delta，将算作没有改进。

Train on 60000 samples, validate on 10000 samples

Epoch 1/15 60000/60000 [==============================] - 10s 173us/sample - loss: 0.2040 - accuracy: 0.9391 - val_loss: 0.1117 - val_accuracy: 0.9648

Epoch 2/15 60000/60000 [==============================] - 9s 150us/sample - loss: 0.0845 - accuracy: 0.9736 - val_loss: 0.0801 - val_accuracy: 0.9748

Epoch 3/15 60000/60000 [==============================] - 9s 151us/sample - loss: 0.0574 - accuracy: 0.9817 - val_loss: 0.0709 - val_accuracy: 0.9795

Epoch 4/15 60000/60000 [==============================] - 9s 149us/sample - loss: 0.0434 - accuracy: 0.9858 - val_loss: 0.0787 - val_accuracy: 0.9761

Epoch 5/15 60000/60000 [==============================] - 9s 151us/sample - loss: 0.0331 - accuracy: 0.9893 - val_loss: 0.0644 - val_accuracy: 0.9808

Epoch 6/15 60000/60000 [==============================] - 9s 150us/sample - loss: 0.0275 - accuracy: 0.9910 - val_loss: 0.0873 - val_accuracy: 0.9779

Epoch 7/15 60000/60000 [==============================] - 9s 151us/sample - loss: 0.0232 - accuracy: 0.9921 - val_loss: 0.0746 - val_accuracy: 0.9805

Epoch 8/15 60000/60000 [==============================] - 9s 151us/sample - loss: 0.0188 - accuracy: 0.9936 - val_loss: 0.1088 - val_accuracy: 0.9748

现在，如果我查看最后三个时期，即第 6、7 和 8 时期，并查看验证损失('val_loss')，它们的值为:

0.0688、0.0843 和 0.0847。

连续3项之间的差异为:0.0155、0.0004。但第一个差异是否大于回调中定义的“min_delta”。

我为 EarlyStopping 编写的代码如下:

# numpy array to hold last 'patience = 3' values-
pv = [0.0688, 0.0843, 0.0847]

# numpy array to compute differences between consecutive elements in 'pv'-
differences = np.diff(pv, n=1)

differences
# array([0.0155, 0.0004])


# minimum change required for monitored metric's improvement-
min_delta = 0.001

# Check whether the consecutive differences is greater than 'min_delta' parameter-
check = differences > min_delta

check
# array([ True,  False])

# Condition to see whether all 3 'val_loss' differences are less than 'min_delta'
# for training to stop since EarlyStopping is called-
if np.all(check == False):
    print("Stop Training - EarlyStopping is called")
    # stop training 

但根据“val_loss”，并非所有最后 3 个 epoch 之间的差异都大于 0.001 的“min_delta”。例如，第一个差值大于 0.001 (0.0843 - 0.0688)，而第二个差值小于 0.001 (0.0847 - 0.0843)。

另外，根据“EarlyStopping”的patience参数定义:

耐心:没有改善的时期数，之后将停止训练。

因此，当“val_loss”连续 3 个周期没有改善时，应该调用 EarlyStopping，其中小于“min_delta”的绝对变化不算作改善。

那为什么要调用EarlyStopping呢？

模型定义和“fit()”的代码是:

import tensorflow_model_optimization as tfmot 
from tensorflow_model_optimization.sparsity import keras as sparsity 
import matplotlib.pyplot as plt from tensorflow.keras.layers import AveragePooling2D, Conv2D
from tensorflow.keras import models, layers, datasets 
from tensorflow.keras.layers import Dense, Flatten, Reshape, Input, InputLayer 
from tensorflow.keras.models import Sequential, Model

# Specify the parameters to be used for layer-wise pruning, NO PRUNING is done here: 
pruning_params_unpruned = {
    'pruning_schedule': sparsity.PolynomialDecay(
        initial_sparsity=0.0, final_sparsity=0.0,
        begin_step = 0, end_step=0, frequency=100) }


def pruned_nn(pruning_params):
    """
    Function to define the architecture of a neural network model
    following 300 100 architecture for MNIST dataset and using
    provided parameter which are used to prune the model.

    Input: 'pruning_params' Python 3 dictionary containing parameters which are used for pruning
    Output: Returns designed and compiled neural network model
    """

    pruned_model = Sequential()
    pruned_model.add(l.InputLayer(input_shape=(784, )))
    pruned_model.add(Flatten())
    pruned_model.add(sparsity.prune_low_magnitude(
        Dense(units = 300, activation='relu', kernel_initializer=tf.initializers.GlorotUniform()),
        **pruning_params))
    # pruned_model.add(l.Dropout(0.2))
    pruned_model.add(sparsity.prune_low_magnitude(
        Dense(units = 100, activation='relu', kernel_initializer=tf.initializers.GlorotUniform()),
        **pruning_params))
    # pruned_model.add(l.Dropout(0.1))
    pruned_model.add(sparsity.prune_low_magnitude(
        Dense(units = num_classes, activation='softmax'),
        **pruning_params))

    # Compile pruned CNN-
    pruned_model.compile(
        loss=tf.keras.losses.categorical_crossentropy,
        # optimizer='adam',
        optimizer=tf.keras.optimizers.Adam(lr = 0.001),
        metrics=['accuracy'])

    return pruned_model


batch_size = 32
epochs = 50



# Instantiate NN- 
orig_model = pruned_nn(pruning_params_unpruned)


# Train unpruned Neural Network-
history_orig = orig_model.fit(
    x = X_train, y = y_train,
    batch_size = batch_size,
    epochs = epochs,
    verbose = 1,
    callbacks = callbacks,
    validation_data = (X_test, y_test),
    shuffle = True )

Answer 1

提前停止回调的行为与以下内容相关:

• 要监控的指标或损失
• min_delta，这是当前周期中监控数量的性能与最佳结果之间被视为改进的最小数量> 在该指标中。
• 耐心，即没有改进的时期的数量(考虑到改进必须比有更大的变化) min_delta)，然后停止算法。

就您而言，最佳的 val_loss 为 0.0644，并且其值应低于 0.0634 才能注册为改进:

Epoch 6/15 val_loss: 0.0873 | Difference is: + 0.0229
Epoch 7/15 val_loss: 0.0746 | Difference is: + 0.0102
Epoch 8/15 val_loss: 0.1088 | Difference is: + 0.0444

请注意，“训练日志”中打印的数量是近似值，您不应根据这些值进行假设。您应该考虑回调或训练历史记录的真实值。

Reference