python - 如何使用 Pytorch 中的预训练权重以 4 个 channel 作为输入修改 resnet 50？

Question

我想改变resnet50，以便我可以切换到4 channel 输入，
对 rgb channel 使用相同的权重，并使用均值为 0 且方差为 0.01 的法线初始化最后一个 channel 。
这是我的代码:

import torch.nn as nn
import torch
from torchvision import models

from misc.layer import Conv2d, FC

import torch.nn.functional as F
from misc.utils import *

import pdb

class Res50(nn.Module):
    def __init__(self,  pretrained=True):
        super(Res50, self).__init__()

        self.de_pred = nn.Sequential(Conv2d(1024, 128, 1, same_padding=True, NL='relu'),
                                     Conv2d(128, 1, 1, same_padding=True, NL='relu'))
        
        self._initialize_weights()

        res = models.resnet50(pretrained=pretrained)
        pretrained_weights = res.conv1.weight

        res.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3,bias=False)

        res.conv1.weight[:,:3,:,:] = pretrained_weights
        res.conv1.weight[:,3,:,:].data.normal_(0.0, std=0.01)
        
        self.frontend = nn.Sequential(
            res.conv1, res.bn1, res.relu, res.maxpool, res.layer1, res.layer2
        )
        
        self.own_reslayer_3 = make_res_layer(Bottleneck, 256, 6, stride=1)        
        self.own_reslayer_3.load_state_dict(res.layer3.state_dict())

        
    def forward(self,x):
        x = self.frontend(x)
        x = self.own_reslayer_3(x)
        x = self.de_pred(x)
        x = F.upsample(x,scale_factor=8)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                m.weight.data.normal_(0.0, std=0.01)
                if m.bias is not None:
                    m.bias.data.fill_(0)
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.fill_(1)
                m.bias.data.fill_(0)

但它产生以下错误，有人有什么建议吗？

/usr/local/lib/python3.6/dist-packages/torch/tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "
Traceback (most recent call last):
  File "train.py", line 62, in <module>
    cc_trainer = Trainer(loading_data,cfg_data,pwd)
  File "/content/drive/My Drive/Folder/Code/trainer.py", line 28, in __init__
    self.optimizer = optim.Adam(self.net.CCN.parameters(), lr=cfg.LR, weight_decay=1e-4) #remenber was 1e-4
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py", line 44, in __init__
    super(Adam, self).__init__(params, defaults)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 51, in __init__
    self.add_param_group(param_group)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 206, in add_param_group
    raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor

Answer 1

理想情况下，ResNet 接受 3 channel 输入。为了使其适用于 4 channel 输入，您必须添加一个额外的层(2D conv)，将 4 channel 输入通过该层，以使该层的输出适用于 ResNet 架构。
脚步

复制模型权重

weight = model.conv1.weight.clone()

为 4 channel 输入添加额外的 2d conv

model.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False) #here 4 indicates 4-channel input

您可以在额外的 con2d 之上添加 Relu 和 BatchNorm。在这个例子中，我没有使用。

将额外的 cov2d 与 ResNet 模型(你之前复制的权重)连接起来

model.conv1.weight[:, :3] = weight
model.conv1.weight[:, 3] = model.conv1.weight[:, 0]

完成

抱歉，我没有修改您的代码。您可以调整代码中的更改。