python - 结合CNN和双向LSTM

Question

我正在尝试结合 CNN 和 LSTM 进行图像分类。
我尝试了以下代码，但出现错误。我有 4 个类(class)，我想对其进行培训和测试。
以下是代码:

from keras.models import Sequential
from keras.layers import LSTM,Conv2D,MaxPooling2D,Dense,Dropout,Input,Bidirectional,Softmax,TimeDistributed


input_shape = (200,300,3)
Model = Sequential()
Model.add(TimeDistributed(Conv2D(
            filters=16, kernel_size=(12, 16), activation='relu', input_shape=input_shape)))
Model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2),strides=2)))
Model.add(TimeDistributed(Conv2D(
            filters=24, kernel_size=(8, 12), activation='relu')))
Model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2),strides=2)))
Model.add(TimeDistributed(Conv2D(
            filters=32, kernel_size=(5, 7), activation='relu')))
Model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2),strides=2)))
Model.add(Bidirectional(LSTM((10),return_sequences=True)))
Model.add(Dense(64,activation='relu'))
Model.add(Dropout(0.5))
Model.add(Softmax(4))
Model.compile(loss='sparse_categorical_crossentropy',optimizer='adam')
Model.build(input_shape)

我收到以下错误:

"Input tensor must be of rank 3, 4 or 5 but was {}.".format(n + 2))ValueError: Input tensor must be of rank 3, 4 or 5 but was 2.

Answer 1

我在代码中发现了很多问题:

您的数据以 4D 形式呈现，如此简单 Conv2D没问题，TimeDistributed不需要

你的输出是 2D 所以设置 return_sequences=False在最后一个 LSTM 单元中

您的最后一层非常困惑:无需在层输出和激活之间放置 dropout

您需要categorical_crossentropy而不是 sparse_categorical_crossentropy因为您的目标是单热编码

LSTM 需要 3D 数据。所以你需要从 4D(卷积的输出)传递到 3D。您可以采用两种可能性:1)进行 reshape (batch_size，H，W * channel)； 2) (batch_size, W, H * channel )。这样，您就可以在 LSTM 中使用 3D 数据

这是一个完整的模型示例:

def ReshapeLayer(x):
    
    shape = x.shape
    
    # 1 possibility: H,W*channel
    reshape = Reshape((shape[1],shape[2]*shape[3]))(x)
    
    # 2 possibility: W,H*channel
    # transpose = Permute((2,1,3))(x)
    # reshape = Reshape((shape[1],shape[2]*shape[3]))(transpose)
    
    return reshape

model = Sequential()
model.add(Conv2D(filters=16, kernel_size=(12, 16), activation='relu', input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2, 2),strides=2))
model.add(Conv2D(filters=24, kernel_size=(8, 12), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),strides=2))
model.add(Conv2D(filters=32, kernel_size=(5, 7), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),strides=2))
model.add(Lambda(ReshapeLayer)) # <========== pass from 4D to 3D
model.add(Bidirectional(LSTM(10, activation='relu', return_sequences=False)))
model.add(Dense(nclasses,activation='softmax'))

model.compile(loss='categorical_crossentropy',optimizer='adam')
model.summary()

here the running notebook