Normalization does not always produce a zero mean. LayerNorm subtracts the mean; RMSNorm adjusts scale using the root mean square directly and generally retains a nonzero mean. Matching output dimensions do not make the two interchangeable in a trained checkpoint.

The missing subtraction matters

For a token hidden vector x of dimension d, RMSNorm uses r=√(Σxᵢ²/d+ε) and yᵢ=gᵢxᵢ/r, with learned componentwise gain g. LayerNorm computes μ=Σxᵢ/d and v=Σ(xᵢ-μ)²/d, then normalizes as (xᵢ-μ)/√(v+ε), optionally adding componentwise gain and bias.

This example omits gain and bias and uses population variance with denominator d. It normalizes hidden dimensions without mixing requests or tokens. Epsilon prevents a zero denominator and affects exact scale invariance.

A constant vector exposes the difference

For x=[3,3], LayerNorm without affine parameters returns [0,0], whereas RMSNorm returns values close to [1,1]. They are not exactly one because epsilon is positive.

For x=[1,2,3], RMSNorm is approximately [0.462910,0.925820,1.388730], whose mean is not zero. LayerNorm is approximately [-1.224744,0,1.224744]. Different outputs do not establish which is more accurate; that requires training and task evidence.

Translation and scaling properties

Ignoring epsilon and affine parameters, LayerNorm is invariant to a uniform translation. RMSNorm generally is not. RMSNorm is invariant to positive rescaling, while negative scaling flips signs.

With positive epsilon, invariance to positive rescaling is approximate rather than exact. For small inputs, epsilon can matter substantially. Tests should include more than large random vectors.

import math

def rmsnorm(x, eps=1e-6):
    denom = math.sqrt(sum(v*v for v in x)/len(x)+eps)
    return [v/denom for v in x]

def layernorm(x, eps=1e-6):
    mean = sum(x)/len(x)
    var = sum((v-mean)**2 for v in x)/len(x)
    return [(v-mean)/math.sqrt(var+eps) for v in x]

x = [1.0, 2.0, 3.0]
r, l = rmsnorm(x), layernorm(x)
assert abs(sum(l)) < 1e-12
assert sum(r)/len(r) > 0.9
assert rmsnorm([0.0,0.0]) == [0.0,0.0]
assert layernorm([3.0,3.0]) == [0.0,0.0]
assert rmsnorm([3.0,3.0])[0] > 0.99
for a,b in zip(rmsnorm(x,0),rmsnorm([2*v for v in x],0)):
    assert math.isclose(a,b)
assert not math.isclose(rmsnorm(x)[0],rmsnorm([v+1 for v in x])[0])
print([round(v,6) for v in r])
print([round(v,6) for v in l])

One fewer step is not a speed guarantee

The code was executed on constant, zero and ordinary vectors, plus positive rescaling with epsilon zero. It trains no model and benchmarks no GPU. Simpler arithmetic does not imply a particular end-to-end throughput improvement.

The RMSNorm paper appeared in 2019. This article checks its definition without generalizing historical experiment speedups to modern LLMs. Deployment also requires checking placement, precision, epsilon and weight shapes. For numerical representation, see LLM quantization.

中文版

Sources

Root Mean Square Layer Normalization

Prepared on October 12, 2026; the article date retains its planned slot.