python - 为什么 Keras 在 model.evaluate、model.predicts 和 model.fit 之间给我不同的结果？

Question

我正在使用基于 resnet50 的双输出模型进行项目。一个输出用于回归任务，第二个输出用于分类任务。

我的主要问题是关于模型评估。在训练期间，我在验证集的两个输出上都取得了不错的结果:
- 综合损失 = 0.32507268732786176
- 准确度 = 0.97375
- Val MSE:4.1454763

model.evaluate 在同一组上给了我以下结果:
- 综合损失 = 0.33064378452301024
- 准确度 = 0.976
- Val MSE = 1.2375486

model.predict 给了我完全不同的结果(我使用 scikit-learn 来计算指标):
- 准确度 = 0.45875
- Val MSE:43.555958365743805
这些最后的值在每次预测执行时都会发生变化。

我在 TF2.0 上工作。
这是我的代码:

valid_generator=datagen.flow_from_dataframe(dataframe=df, 
                                            directory=PATH, 
                                            x_col="X", 
                                            y_col=["yReg","yCls"],  
                                            class_mode="multi_output", 
                                            target_size=(IMG_SIZE, IMG_SIZE), 
                                            batch_size=batch_size,
                                            subset="validation",
                                            shuffle=False,
                                            workers = 0)
def generate_data_generator(generator, train=True):
    while True:
        Xi, yi = train_generator.next()
        y2 = []
        for e in yi[1]:
            y2.append(to_categorical(e, 7))
        y2 = np.array(y2)
        if train: # Augmentation for training only
            Xi = Xi.astype('uint8')
            Xi_aug = seq(images=Xi) # imgaug lib needs uint8
            Xi_aug = Xi_aug.astype('float32')
            Xi_aug = preprocess_input(Xi_aug) # resnet50 preprocessing
            yield Xi_aug, [yi[0], y2]
        else: # Validation
            yield preprocess_input(Xi), [yi[0], y2]


model.fit_generator(generator=generate_data_generator(train_generator, True),
    steps_per_epoch=STEP_SIZE_TRAIN,
    validation_data=generate_data_generator(valid_generator, False),
    validation_steps=STEP_SIZE_VALID,
    verbose=1, 
    epochs=50, 
    callbacks=[checkpoint, tfBoard],
    )

evalu = model.evaluate_generator(generate_data_generator(valid_generator, False), steps=STEP_SIZE_VALID)
print(model.metrics_names)
print(evalu)

preds = model.predict_generator(generate_data_generator(valid_generator, False), steps=STEP_SIZE_VALID, workers = 0)
labels = valid_generator.labels

print("MSE error:", me.mean_squared_error(labels[0], preds[0]))
print("Accuracy:", me.accuracy_score(labels[1], preds[1].argmax(axis=1)))

我究竟做错了什么 ？

谢谢您的帮助 !

Answer 1

您仅使用一个数据点计算准确度 labels[1], preds[1]而不是所有数据点。您需要考虑所有数据点来计算准确度，以便将结果与 model.evaluate_generator 进行比较。 .您还计算了 MSE在 labels[0], preds[0]数据点，但准确性在 labels[1], preds[1] 上数据点。考虑这两种情况下的所有数据点。

下面是一个二元分类的例子，我没有对验证数据做任何数据增强。您可以在没有增强的情况下构建验证生成器并设置 shuffle=False每次生成相同批次的数据，因此您将获得相同的结果 model.evaluate_generator和 model.predict_generator .

验证生成器 -

validation_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our validation data

val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,
                                                              directory=validation_dir,
                                                              shuffle=False,
                                                              seed=10,
                                                              target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                              class_mode='binary')

以下是所有匹配 的准确率结果——

model.fit_generator

history = model.fit_generator(
          train_data_gen,
          steps_per_epoch=total_train // batch_size,
          epochs=5,
          validation_data=val_data_gen,
          validation_steps=total_val // batch_size)

输出 -

Found 2000 images belonging to 2 classes.
Found 1000 images belonging to 2 classes.
Epoch 1/5
20/20 [==============================] - 27s 1s/step - loss: 0.8691 - accuracy: 0.4995 - val_loss: 0.6850 - val_accuracy: 0.5000
Epoch 2/5
20/20 [==============================] - 26s 1s/step - loss: 0.6909 - accuracy: 0.5145 - val_loss: 0.6880 - val_accuracy: 0.5000
Epoch 3/5
20/20 [==============================] - 26s 1s/step - loss: 0.6682 - accuracy: 0.5345 - val_loss: 0.6446 - val_accuracy: 0.6320
Epoch 4/5
20/20 [==============================] - 26s 1s/step - loss: 0.6245 - accuracy: 0.6180 - val_loss: 0.6214 - val_accuracy: 0.5920
Epoch 5/5
20/20 [==============================] - 27s 1s/step - loss: 0.5696 - accuracy: 0.6795 - val_loss: 0.6468 - val_accuracy: 0.6270

model.evaluate_generator

evalu = model.evaluate_generator(val_data_gen)
print(model.metrics_names)
print(evalu)

输出 -

['loss', 'accuracy']
[0.646793782711029, 0.6269999742507935]

model.predict_generator

from sklearn.metrics import mean_squared_error, accuracy_score
preds = model.predict_generator(val_data_gen)
y_pred = tf.where(preds<=0.5,0,1)

labels = val_data_gen.labels
y_true = labels

# confusion_matrix(y_true, y_pred)
print("Accuracy:", accuracy_score(y_true, y_pred))

输出 -

Accuracy: 0.627

完整代码供您引用 -

%tensorflow_version 2.x
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.optimizers import Adam

import os
import numpy as np
import matplotlib.pyplot as plt

_URL = 'https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip'

path_to_zip = tf.keras.utils.get_file('cats_and_dogs.zip', origin=_URL, extract=True)

PATH = os.path.join(os.path.dirname(path_to_zip), 'cats_and_dogs_filtered')

train_dir = os.path.join(PATH, 'train')
validation_dir = os.path.join(PATH, 'validation')

train_cats_dir = os.path.join(train_dir, 'cats')  # directory with our training cat pictures
train_dogs_dir = os.path.join(train_dir, 'dogs')  # directory with our training dog pictures
validation_cats_dir = os.path.join(validation_dir, 'cats')  # directory with our validation cat pictures
validation_dogs_dir = os.path.join(validation_dir, 'dogs')  # directory with our validation dog pictures

num_cats_tr = len(os.listdir(train_cats_dir))
num_dogs_tr = len(os.listdir(train_dogs_dir))

num_cats_val = len(os.listdir(validation_cats_dir))
num_dogs_val = len(os.listdir(validation_dogs_dir))

total_train = num_cats_tr + num_dogs_tr
total_val = num_cats_val + num_dogs_val

batch_size = 100
epochs = 5
IMG_HEIGHT = 150
IMG_WIDTH = 150

train_image_generator = ImageDataGenerator(rescale=1./255,brightness_range=[0.5,1.5]) # Generator for our training data
validation_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our validation data

train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
                                                           directory=train_dir,
                                                           shuffle=True,
                                                           target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                           class_mode='binary')

val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,
                                                              directory=validation_dir,
                                                              shuffle=False,
                                                              seed=10,
                                                              target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                              class_mode='binary')

model = Sequential([
    Conv2D(16, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
    MaxPooling2D(),
    Conv2D(32, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Conv2D(64, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Flatten(),
    Dense(512, activation='relu'),
    Dense(1)
])

model.compile(optimizer="adam", 
          loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
          metrics=['accuracy'])

history = model.fit_generator(
          train_data_gen,
          steps_per_epoch=total_train // batch_size,
          epochs=epochs,
          validation_data=val_data_gen,
          validation_steps=total_val // batch_size)

evalu = model.evaluate_generator(val_data_gen, steps=total_val // batch_size)
print(model.metrics_names)
print(evalu)

from sklearn.metrics import mean_squared_error, accuracy_score
#val_data_gen.reset()
preds = model.predict_generator(val_data_gen, steps=total_val // batch_size)
y_pred = tf.where(preds<=0.5,0,1)

labels = val_data_gen.labels
y_true = labels

test_labels = []

for i in range(0,10):
    test_labels.extend(np.array(val_data_gen[i][1]))

# confusion_matrix(y_true, y_pred)
print("Accuracy:", accuracy_score(test_labels, y_pred))

还要记住， fit_generator , evaluate_generator和 predict_generator功能已弃用。它将在 future 版本中删除。更新说明:请分别使用支持生成器的Model.fit、Model.evaluate、Model.predict。

希望这能回答你的问题。快乐学习。