python - Keras fit_generator 的预测与 fit 不同

Question

我正在尝试在 Keras 中训练用于图像分割 (U-Net) 的模型，并首先生成两个包含我的训练(和验证)图像和掩模的列表。然后我训练了模型，如下所示。

x_train_val = # list of images (nr_images, 256, 256, 3)
y_train_val = # list of masks (nr_images, 256, 256, 1)

# Define model
def standard_unet():
    inputs = Input((img_size, img_size, 3))

    s = Lambda(lambda x: x / 255) (inputs)

    c1 = Conv2D(8, (3, 3), activation='relu', padding='same') (inputs)
    c1 = Conv2D(8, (3, 3), activation='relu', padding='same') (c1)
    p1 = MaxPooling2D((2, 2)) (c1)

    c2 = Conv2D(16, (3, 3), activation='relu', padding='same') (p1)
    c2 = Conv2D(16, (3, 3), activation='relu', padding='same') (c2)
    p2 = MaxPooling2D((2, 2)) (c2)

    c3 = Conv2D(32, (3, 3), activation='relu', padding='same') (p2)
    c3 = Conv2D(32, (3, 3), activation='relu', padding='same') (c3)
    p3 = MaxPooling2D((2, 2)) (c3)

    c4 = Conv2D(64, (3, 3), activation='relu', padding='same') (p3)
    c4 = Conv2D(64, (3, 3), activation='relu', padding='same') (c4)
    p4 = MaxPooling2D(pool_size=(2, 2)) (c4)

    c5 = Conv2D(128, (3, 3), activation='relu', padding='same') (p4)
    c5 = Conv2D(128, (3, 3), activation='relu', padding='same') (c5)

    u6 = Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same') (c5)
    u6 = concatenate([u6, c4])
    c6 = Conv2D(64, (3, 3), activation='relu', padding='same') (u6)
    c6 = Conv2D(64, (3, 3), activation='relu', padding='same') (c6)

    u7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)
    u7 = concatenate([u7, c3])
    c7 = Conv2D(32, (3, 3), activation='relu', padding='same') (u7)
    c7 = Conv2D(32, (3, 3), activation='relu', padding='same') (c7)

    u8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)
    u8 = concatenate([u8, c2])
    c8 = Conv2D(16, (3, 3), activation='relu', padding='same') (u8)
    c8 = Conv2D(16, (3, 3), activation='relu', padding='same') (c8)

    u9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)
    u9 = concatenate([u9, c1], axis=3)
    c9 = Conv2D(8, (3, 3), activation='relu', padding='same') (u9)
    c9 = Conv2D(8, (3, 3), activation='relu', padding='same') (c9)

    outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)

    model = Model(inputs=[inputs], outputs=[outputs])

    return model

# IoU metric
def mean_iou(y_true, y_pred):
    prec = []
    for t in np.arange(0.5, 1.0, 0.05):
        y_pred_ = tf.to_int32(y_pred > t)
        score, up_opt = tf.metrics.mean_iou(y_true, y_pred_, 2)
        K.get_session().run(tf.local_variables_initializer())
        with tf.control_dependencies([up_opt]):
            score = tf.identity(score)
        prec.append(score)
    return K.mean(K.stack(prec), axis=0)

# Dice coef loss
def dice_coef(y_true, y_pred):
    smooth = 1.
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

def bce_dice_loss(y_true, y_pred):
    return 0.5 * binary_crossentropy(y_true, y_pred) - dice_coef(y_true, y_pred)

# Model compiling
K.clear_session()
model = standard_unet()
model.compile(optimizer='adam', loss=bce_dice_loss, metrics=[mean_iou])

# Fitting
model.fit(x_train_val, y_train_val, validation_split=0.1, epochs=20)

这完全按照预期工作，当我尝试对测试图像进​​行预测时，我得到了不错的结果。由于我想增加训练图像的数量，我尝试使用 ImageDataGenerator 和 train_generator 使用以下函数。

# Runtime data augmentation
def get_train_test_augmented(x_data=x_train_val, y_data=y_train_val, validation_split=0.1, batch_size=32):
    x_train, x_valid, y_train, y_valid = train_test_split(x_data, y_data,
                                                          train_size=1-validation_split,
                                                          test_size=validation_split)

    data_gen_args = dict(rotation_range=45.,
                         width_shift_range=0.1,
                         height_shift_range=0.1,
                         horizontal_flip=True,
                         vertical_flip=True,
                         fill_mode='reflect')  #use 'constant'??

    x_datagen = ImageDataGenerator(**data_gen_args)
    y_datagen = ImageDataGenerator(**data_gen_args)
    x_datagen.fit(x_train, augment=True)
    y_datagen.fit(y_train, augment=True)
    x_train_augmented = x_datagen.flow(x_train, batch_size=batch_size, shuffle=True)
    y_train_augmented = y_datagen.flow(y_train, batch_size=batch_size, shuffle=True)

    # combine generators into one which yields image and masks
    train_generator = zip(x_train_augmented, y_train_augmented)

    return train_generator

对这些图像的目视检查表明，它们包含我所期望的内容(增强图像和蒙版)。然而，当我现在拟合我的模型时，我的预测总是空白。

train_generator = get_train_test_augmented()
model.fit_generator(train_generator, epochs=20)

有人在空白图像预测方面遇到过同样的问题或知道如何解决它吗？谢谢，BBQuercus。

Answer 1

您正在使用图像和掩模生成器分别进行图像生成，将会发生的情况是输入图像和标签(掩模)的随机变换将不相同。更重要的是，您正在打乱两个生成器，因此它们甚至不相互对应(适合生成器中的图像和蒙版)。

This github issue comment还讨论了这一点，并建议创建一个额外的生成器来合并两者。

尝试使用相同的种子为两个生成器播种，看看它是否会改变任何东西。

<小时/>

编辑

当我进行图像去噪时，我注意到使用 zip 的解决方案不是最优的，因为在拟合时无法使用 use_multiprocessing=True 。解决方案是实现自定义生成器合并:

class MergedGenerators(Sequence):
    def __init__(self, *generators):
        self.generators = generators
        # TODO add a check to verify that all generators have the same length

    def __len__(self):
        return len(self.generators[0])

    def __getitem__(self, index):
        return [generator[index] for generator in self.generators]

train_generator = MergedGenerators(image_generator, mask_generator)