python - 比较 Conv2D 与 Tensorflow 和 PyTorch 之间的填充

Question

我正在尝试将从 Tensorflow 模型保存的权重导入 PyTorch。到目前为止，结果非常相似。当模型使用 stride=2 调用 conv2d 时，我遇到了麻烦。

为了验证不匹配，我在 TF 和 PyTorch 之间建立了一个非常简单的比较。首先，我将 conv2d 与 stride=1 进行比较。

import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F


np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
# PyTorch adopts [outputC, inputC, kH, kW]
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow defaults to NHWC
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 1, 1, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=1)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))

这次执行的结果是:

Mean of Abs Diff: 2.0443112092038973e-08

当我将 stride 更改为 2 时，结果开始发生变化。

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=2)

这次执行的结果是:

Mean of Abs Diff: 0.2104552686214447

根据 PyTorch 文档，conv2d uses zero-padding由 padding 参数定义。因此，在我的示例中，零被添加到输入的左侧、顶部、右侧和底部。

如果 PyTorch 只是简单地根据输入参数在两侧添加填充，它应该很容易在 Tensorflow 中复制。

# Manually add padding - consistent with PyTorch
paddings = tf.constant([[0, 0], [1, 1], [1, 1], [0, 0]])
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_tf = tf.pad(x_tf, paddings, "CONSTANT")

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="VALID")

这个比较的结果是:

Mean of Abs Diff: 1.6035047067930464e-08

这告诉我的是，如果我能够以某种方式将默认填充行为从 Tensorflow 复制到 PyTorch，那么我的结果将是相似的。

This question检查了 Tensorflow 中填充的行为。 TF documentation explains how padding is added for "SAME" convolutions.我在写这个问题时发现了这些链接。

既然我知道了Tensorflow的padding策略，我就可以在PyTorch中实现了。

Answer 1

为了复制行为，填充大小按照 Tensorflow 文档中的描述进行计算。在这里，我通过设置 stride=2 并填充 PyTorch 输入来测试填充行为。

import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F


np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow padding behavior. Assuming that kH == kW to keep this simple.
stride = 2
if x.shape[2] % stride == 0:
    pad = max(weights.shape[0] - stride, 0)
else:
    pad = max(weights.shape[0] - (x.shape[2] % stride), 0)

if pad % 2 == 0:
    pad_val = pad // 2
    padding = (pad_val, pad_val, pad_val, pad_val)
else:
    pad_val_start = pad // 2
    pad_val_end = pad - pad_val_start
    padding = (pad_val_start, pad_val_end, pad_val_start, pad_val_end)

x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)
x_torch = F.pad(x_torch, padding, "constant", 0)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, stride, stride, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=0, stride=stride)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))

输出是:

Mean of Abs Diff: 2.2477470551507395e-08

当我开始写这个问题时，我不太确定为什么会发生这种情况，但一些阅读很快就澄清了这一点。我希望这个例子可以帮助其他人。