machine-learning - pytorch中的层标准化？

Question

x = torch.tensor([[1.5,0,0,0,0]]) 的层归一化不应该是 [[1.5,-0.5,-0.5 ,-0.5]] ？根据本文paper以及 pytorch doc 中的方程。但 torch.nn.LayerNorm 给出 [[ 1.7320, -0.5773, -0.5773, -0.5773]]

这里是示例代码:

x = torch.tensor([[1.5,.0,.0,.0]])
layerNorm = torch.nn.LayerNorm(4, elementwise_affine = False)

y1 = layerNorm(x)

mean = x.mean(-1, keepdim = True)
var = x.var(-1, keepdim = True)
y2 = (x-mean)/torch.sqrt(var+layerNorm.eps)

地点:

y1 == tensor([[ 1.7320, -0.5773, -0.5773, -0.5773]])
y2 == tensor([[ 1.5000, -0.5000, -0.5000, -0.5000]])

Answer 1

使用裸 PyTorch 的 Layer Norm 层的另一个简化实现。

from typing import Tuple
import torch


def layer_norm(
    x: torch.Tensor, dim: Tuple[int], eps: float = 0.00001
) -> torch.Tensor:
    mean = torch.mean(x, dim=dim, keepdim=True)
    var = torch.square(x - mean).mean(dim=dim, keepdim=True)
    return (x - mean) / torch.sqrt(var + eps)


def test_that_results_match() -> None:
    dims = (1, 2)
    X = torch.normal(0, 1, size=(3, 3, 3))

    indices = torch.tensor(dims)
    normalized_shape = torch.tensor(X.size()).index_select(0, indices)
    orig_layer_norm = torch.nn.LayerNorm(normalized_shape)

    y = orig_layer_norm(X)
    y_hat = layer_norm(X, dim=dims)

    assert torch.allclose(y, y_hat)

请注意，原始实现还具有可训练参数 𝛄 和 β(请参阅 docs )。