tensorflow - 在 Keras/TensorFlow 中训练一个具有可变大小输入的完全卷积神经网络需要花费不合理的长时间

Question

我正在尝试为图像分类实现一个 FCNN，它可以接受可变大小的输入。该模型是在 Keras 中构建的，带有 TensorFlow 后端。

考虑以下玩具示例:

model = Sequential()

# width and height are None because we want to process images of variable size 
# nb_channels is either 1 (grayscale) or 3 (rgb)
model.add(Convolution2D(32, 3, 3, input_shape=(nb_channels, None, None), border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(32, 3, 3, border_mode='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(16, 1, 1))
model.add(Activation('relu'))

model.add(Convolution2D(8, 1, 1))
model.add(Activation('relu'))

# reduce the number of dimensions to the number of classes
model.add(Convolution2D(nb_classses, 1, 1))
model.add(Activation('relu'))

# do global pooling to yield one value per class
model.add(GlobalAveragePooling2D())

model.add(Activation('softmax'))

该模型运行良好，但我遇到了性能问题。与固定尺寸输入的训练相比，可变尺寸图像的训练花费的时间长得不合理。如果我将所有图像的大小调整为数据集中的最大大小，则训练模型所需的时间仍远少于对可变大小输入进行训练的时间。 input_shape=(nb_channels, None, None)也是如此指定可变大小输入的正确方法？有什么办法可以缓解这个性能问题？

更新
model.summary()对于具有 3 个类和灰度图像的模型:

Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
convolution2d_1 (Convolution2D)  (None, 32, None, None 320         convolution2d_input_1[0][0]      
____________________________________________________________________________________________________
activation_1 (Activation)        (None, 32, None, None 0           convolution2d_1[0][0]            
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)    (None, 32, None, None 0           activation_1[0][0]               
____________________________________________________________________________________________________
convolution2d_2 (Convolution2D)  (None, 32, None, None 9248        maxpooling2d_1[0][0]             
____________________________________________________________________________________________________
maxpooling2d_2 (MaxPooling2D)    (None, 32, None, None 0           convolution2d_2[0][0]            
____________________________________________________________________________________________________
convolution2d_3 (Convolution2D)  (None, 16, None, None 528         maxpooling2d_2[0][0]             
____________________________________________________________________________________________________
activation_2 (Activation)        (None, 16, None, None 0           convolution2d_3[0][0]            
____________________________________________________________________________________________________
convolution2d_4 (Convolution2D)  (None, 8, None, None) 136         activation_2[0][0]               
____________________________________________________________________________________________________
activation_3 (Activation)        (None, 8, None, None) 0           convolution2d_4[0][0]            
____________________________________________________________________________________________________
convolution2d_5 (Convolution2D)  (None, 3, None, None) 27          activation_3[0][0]               
____________________________________________________________________________________________________
activation_4 (Activation)        (None, 3, None, None) 0           convolution2d_5[0][0]            
____________________________________________________________________________________________________
globalaveragepooling2d_1 (Global (None, 3)             0           activation_4[0][0]               
____________________________________________________________________________________________________
activation_5 (Activation)        (None, 3)             0           globalaveragepooling2d_1[0][0]   
====================================================================================================
Total params: 10,259
Trainable params: 10,259
Non-trainable params: 0

Answer 1

我认为@marcin-możejko 在他的评论中可能有正确的答案。
可能与this bug有关，这是刚刚修复的。和 this patch如果编译过于频繁，可能会警告您。

所以升级到 tf-nightly-gpu-2.0-preview 包可能会解决这个问题。
您是否也遇到过 tf.keras 的问题？ .

If I resize all images to the maximum size in the data set it still takes far less time to train the model than training on the variable size input

请注意，对于具有“相同”填充的基本卷积，除了像素对齐之外，零填充应该对输出具有“无”影响。

因此，一种方法是在固定的尺寸列表上进行训练，并将这些尺寸的零垫图像设为零。例如，训练 128x128、256x256、512x512 的批次。如果你不能修复动态编译的东西，这至少只会编译 3 次。这有点像有时在序列模型中看到的二维“按序列长度”方法。