python - 如何将具有像 ResNet 这样的非序列架构的 Keras 模型拆分为子模型？

Question

我的模型是 resnet-152，我想将其切成两个子模型，问题是第二个子模型，我不知道如何构建从中间层到输出的模型

我尝试了 this response 中的代码它对我不起作用，这是我的代码:

def getLayerIndexByName(model, layername):
    for idx, layer in enumerate(model.layers):
        if layer.name == layername:
            return idx

idx = getLayerIndexByName(resnet, 'res3a_branch2a')

input_shape = resnet.layers[idx].get_input_shape_at(0) # which is here in my case (None, 55, 55, 256)

layer_input = Input(shape=input_shape[1:]) # as keras will add the batch shape

# create the new nodes for each layer in the path
x = layer_input
for layer in resnet.layers[idx:]:
    x = layer(x)

# create the model
new_model = Model(layer_input, x)

我收到此错误:

ValueError: Input 0 is incompatible with layer res3a_branch1: expected axis -1 of input shape to have value 256 but got shape (None, 28, 28, 512).

我也尝试过这个功能:

def split(model, start, end):
    confs = model.get_config()
    kept_layers = set()
    for i, l in enumerate(confs['layers']):
        if i == 0:
            confs['layers'][0]['config']['batch_input_shape'] = model.layers[start].input_shape
            if i != start:
                confs['layers'][0]['name'] += str(random.randint(0, 100000000)) # rename the input layer to avoid conflicts on merge
                confs['layers'][0]['config']['name'] = confs['layers'][0]['name']
        elif i < start or i > end:
            continue
        kept_layers.add(l['name'])
    # filter layers
    layers = [l for l in confs['layers'] if l['name'] in kept_layers]
    layers[1]['inbound_nodes'][0][0][0] = layers[0]['name']
    # set conf
    confs['layers'] = layers
    confs['input_layers'][0][0] = layers[0]['name']
    confs['output_layers'][0][0] = layers[-1]['name']
    # create new model
    submodel = Model.from_config(confs)
    for l in submodel.layers:
        orig_l = model.get_layer(l.name)
        if orig_l is not None:
            l.set_weights(orig_l.get_weights())
    return submodel

我收到此错误:

ValueError: Unknown layer: Scale

因为我的 resnet152 包含一个 Scale 层。

这是一个工作版本:

import resnet   # pip install resnet
from keras.models import Model
from keras.layers import Input

def getLayerIndexByName(model, layername):
    for idx, layer in enumerate(model.layers):
        if layer.name == layername:
            return idx


resnet = resnet.ResNet152(weights='imagenet')

idx = getLayerIndexByName(resnet, 'res3a_branch2a')

model1 = Model(inputs=resnet.input, outputs=resnet.get_layer('res3a_branch2a').output)

input_shape = resnet.layers[idx].get_input_shape_at(0) # get the input shape of desired layer
print(input_shape[1:])
layer_input = Input(shape=input_shape[1:]) # a new input tensor to be able to feed the desired layer

# create the new nodes for each layer in the path
x = layer_input
for layer in resnet.layers[idx:]:
    x = layer(x)

# create the model
model2 = Model(layer_input, x)

model2.summary()

这是错误:

ValueError: Input 0 is incompatible with layer res3a_branch1: expected axis -1 of input shape to have value 256 but got shape (None, 28, 28, 512)

Answer 1

正如我在评论部分提到的，由于 ResNet 模型没有线性架构(即它具有跳过连接，并且一个层可能连接到多个层)，因此您不能简单地遍历模型的各个层循环中的另一个层之后，并对循环中前一层的输出应用一层(即与 this method works 的线性架构模型不同)。

因此，您需要找到各层的连通性并遍历该连通性图，以便能够构建原始模型的子模型。目前，我想到了这个解决方案:

1. 指定子模型的最后一层。
2. 从该层开始，找到与其连接的所有层。
3. 获取这些连接层的输出。
4. 对收集的输出应用最后一层。

显然，步骤#3意味着递归:为了获得连接层(即X)的输出，我们首先需要找到它们的连接层(即Y)，获取它们的输出(即Y的输出)，然后将它们应用到这些输出(即将 X 应用于 Y 的输出)。此外，要找到连接层，您需要了解一些 Keras 的内部结构，这已在 this answer 中进行了介绍。 。所以我们想出了这个解决方案:

from keras.applications.resnet50 import ResNet50
from keras import models
from keras import layers

resnet = ResNet50()

# this is the split point, i.e. the starting layer in our sub-model
starting_layer_name = 'activation_46'

# create a new input layer for our sub-model we want to construct
new_input = layers.Input(batch_shape=resnet.get_layer(starting_layer_name).get_input_shape_at(0))

layer_outputs = {}
def get_output_of_layer(layer):
    # if we have already applied this layer on its input(s) tensors,
    # just return its already computed output
    if layer.name in layer_outputs:
        return layer_outputs[layer.name]

    # if this is the starting layer, then apply it on the input tensor
    if layer.name == starting_layer_name:
        out = layer(new_input)
        layer_outputs[layer.name] = out
        return out

    # find all the connected layers which this layer
    # consumes their output
    prev_layers = []
    for node in layer._inbound_nodes:
        prev_layers.extend(node.inbound_layers)

    # get the output of connected layers
    pl_outs = []
    for pl in prev_layers:
        pl_outs.extend([get_output_of_layer(pl)])

    # apply this layer on the collected outputs
    out = layer(pl_outs[0] if len(pl_outs) == 1 else pl_outs)
    layer_outputs[layer.name] = out
    return out

# note that we start from the last layer of our desired sub-model.
# this layer could be any layer of the original model as long as it is
# reachable from the starting layer
new_output = get_output_of_layer(resnet.layers[-1])

# create the sub-model
model = models.Model(new_input, new_output)

重要说明:

1. 此解决方案假设原始模型中的每个层仅使用一次，即它不适用于暹罗网络，其中一个层可以共享，因此可能在不同的输入张量上应用多次。

2. 如果您希望将一个模型正确拆分为多个子模型，那么仅使用这些层作为拆分点是有意义的(例如上面的 starting_layer_name 所示)代码)不在分支中(例如，在 ResNet 中，合并层之后的激活层是一个不错的选择，但您选择的 res3a_branch2a 不是一个好的选择，因为它位于分支中)。为了更好地了解模型的原始架构，您始终可以使用 plot_model() 实用函数绘制其图表:

   from keras.applications.resnet50 import ResNet50
   from keras.utils import plot_model
   
   resnet = ResNet50()
   plot_model(model, to_file='resnet_model.png')
   

3. 由于在构建子模型后会创建新节点，因此不要尝试构建另一个有重叠的子模型(即，如果没有重叠，就可以了! ) 与之前的子模型在上面代码的同一运行中；否则，您可能会遇到错误。