python - 如何在 Pytorch 中实现上限 JSD 损失？

Question

我正在尝试“复制”TextGAN使用 pytorch，我是 pytorch 的新手。我目前关心的是复制 L_G(第 3 页等式 7)，这是我当前的代码:

class JSDLoss(nn.Module):

    def __init__(self):
        super(JSDLoss,self).__init__()

    def forward(self, batch_size, f_real, f_synt):
        assert f_real.size()[1] == f_synt.size()[1]

        f_num_features = f_real.size()[1]
        identity = autograd.Variable(torch.eye(f_num_features)*0.1, requires_grad=False)

        if use_cuda:
            identity = identity.cuda(gpu)

        f_real_mean = torch.mean(f_real, 0, keepdim=True)
        f_synt_mean = torch.mean(f_synt, 0, keepdim=True)

        dev_f_real = f_real - f_real_mean.expand(batch_size,f_num_features)
        dev_f_synt = f_synt - f_synt_mean.expand(batch_size,f_num_features)

        f_real_xx = torch.mm(torch.t(dev_f_real), dev_f_real)
        f_synt_xx = torch.mm(torch.t(dev_f_synt), dev_f_synt)

        cov_mat_f_real = (f_real_xx / batch_size) - torch.mm(f_real_mean, torch.t(f_real_mean)) + identity
        cov_mat_f_synt = (f_synt_xx / batch_size) - torch.mm(f_synt_mean, torch.t(f_synt_mean)) + identity

        cov_mat_f_real_inv = torch.inverse(cov_mat_f_real)
        cov_mat_f_synt_inv = torch.inverse(cov_mat_f_synt)

        temp1 = torch.trace(torch.add(torch.mm(cov_mat_f_synt_inv, cov_mat_f_real), torch.mm(cov_mat_f_real_inv, cov_mat_f_synt)))
        temp1 = temp1.view(1,1)
        temp2 = torch.mm(torch.mm((f_synt_mean - f_real_mean), (cov_mat_f_synt_inv + cov_mat_f_real_inv)), torch.t(f_synt_mean - f_real_mean))
        loss_g = torch.add(temp1, temp2).mean()

        return loss_g

它有效。但是，我怀疑这不是创建自定义损失的方法。非常感谢任何形式的帮助!提前致谢:)

Answer 1

如何在 Pytorch 中创建自定义损失

这就是您在 Pytorch 中创建自定义损失的方式。您需要满足以下要求:

• The value finally being returned by a loss function MUST BE a scalar value. Not a vector/tensor.
• The value being returned must be a Variable. This is so that it can be used to update the parameters in your model. The best way to do so is to just make sure that both x and y being passed in are Variables. That way any function of the two will also be a Variable.
• Define the __init__ and forward methods

您可以在 Pytorch 源代码中找到几个可用作示例的损失模块:https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py

如果您将小批量张量传递给损失函数，则无需将小批量大小传递给 forward 函数，因为大小可以在 forward 函数。

如何使用自定义损失

实现损失函数后，您可以按如下方式使用它，例如:

loss = YourLoss()
input = autograd.Variable(torch.randn(3, 5), requires_grad=True)
target = autograd.Variable(torch.randn(3, 5))
output = loss(input, target)
output.backward()

loss.backward() 为网络中的每个参数 x 计算 dloss/dx，其中 requires_grad=True。对于每个参数 x，这些都会累积到 x.grad 中。在伪代码中:

x.grad += dloss/dx

optimizer.step 使用梯度 x.grad 更新 x 的值。例如，SGD 优化器执行:

x += -lr * x.grad

optimizer.zero_grad() 清除优化器中每个参数 x 的 x.grad。在 loss.backward() 之前调用它很重要，否则您将累积多次遍历的梯度。