Epsilon in RMSNorm is not a small number that can be copied without further thought. It is added to the mean square of the input, preventing a zero denominator while also changing the output of small inputs. Preserve an existing model’s configuration first; then inspect the ratio between input energy and epsilon to understand its effect. A finite output alone does not establish migration parity.

This follows the site’s RMSNorm versus LayerNorm article with a narrower question: which inputs reveal epsilon differences, and how can reproducible numerical checks expose them? 阅读中文版.

Fix the definition: epsilon is inside the square root

For a d-dimensional vector, let q be the mean of its squared elements. This article uses the following definition, temporarily setting the learned elementwise gain g to one. The PyTorch RMSNorm documentation likewise places epsilon inside the square root with the mean square.

Epsilon therefore shares the numerical scale of q, rather than being added directly to its square root. Replacing the denominator with √q+epsilon produces a different function. Two implementations that both contain a square root and an epsilon addition are not necessarily equivalent. Write out the parentheses before comparing configuration names.

The derivation below concerns the normalization step, not training quality. The original RMSNorm paper was published in 2019. This is an explanation of a numerical mechanism, not a newly announced model result.

One ratio reveals the size of the effect

Assume q is positive. Without epsilon, the normalized output has an RMS of one. With epsilon, every component receives an additional common attenuation factor A. Dividing the two denominators gives:

The relevant question is how large epsilon is relative to q. When q is 100 times epsilon, A is approximately 0.995. When they are equal, A is about 0.707. When q is one hundredth of epsilon, A falls to about 0.0995. The first change is small; the last should not be described as nearly unchanged.

The chart also gives output RMS before learned gain. If gain differs across components, the final output RMS need not equal A. Do not apply this intermediate result to the complete module without that qualification. When q is zero, every input component is zero and positive epsilon yields a zero output. Comparing that result to an epsilon-free 0/0 would be undefined.

Positive scaling is no longer exactly invariant

Scale the entire input by a positive c. The mean square becomes c²q. Cancelling c between numerator and denominator leaves xᵢ/√(q+epsilon/c²). Holding epsilon fixed while scaling the input is therefore equivalent to changing its relative strength.

For large inputs, epsilon usually has a weaker relative effect. Shrink the input sufficiently and it can dominate the denominator. The statement that RMSNorm is scale invariant needs assumptions such as ignoring epsilon. During a numerical port, near-zero vectors are more likely to expose this difference than random vectors at an ordinary scale.

This explains a misleading acceptance result: close outputs on ordinary samples do not establish parity over every relevant input range. A test set with no small-magnitude examples has not examined the part of the function where epsilon differences are especially visible.

A runnable reference for small and large inputs

The following scalar Python reference explicitly uses epsilon=1e-6. It combines terms with hypot to avoid directly squaring the enormous values in the example and obtaining infinity. It checks a formula and finite cases; it is neither a high-performance tensor kernel nor a simulation of half-precision arithmetic.

import math

def rmsnorm_reference(x, eps=1e-6):
    x = [float(v) for v in x]
    if not x or not all(math.isfinite(v) for v in x):
        raise ValueError("finite, nonempty input required")
    if not math.isfinite(eps) or eps <= 0:
        raise ValueError("eps must be finite and positive")
    # Scalar teaching reference: avoid directly squaring large values.
    root_d = math.sqrt(len(x))
    denom = math.hypot(*(v / root_d for v in x), math.sqrt(eps))
    return [v / denom for v in x]

def attenuation(q_over_eps):
    return math.sqrt(q_over_eps / (q_over_eps + 1.0))

for ratio in [0.01, 0.1, 1.0, 10.0, 100.0]:
    x = [math.sqrt(ratio * 1e-6)] * 2
    y = rmsnorm_reference(x)
    expected = attenuation(ratio)
    assert math.isclose(y[0], expected, rel_tol=1e-12)
    print(ratio, round(y[0], 6))
assert rmsnorm_reference([0, 0]) == [0.0, 0.0]
assert math.isclose(rmsnorm_reference([1e200, -1e200])[0], 1.0)
a = rmsnorm_reference([1e-3, 1e-3], eps=1e-6)[0]
b = rmsnorm_reference([1e-3, 1e-3], eps=1e-4)[0]
assert math.isclose(a, 1 / math.sqrt(2))
assert math.isclose(b, 1 / math.sqrt(101))
for bad, eps in [([], 1e-6), ([float("nan")], 1e-6),
                 ([1.0], 0), ([1.0], -1e-6)]:
    try:
        rmsnorm_reference(bad, eps)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid case was accepted")

# Illustrative 1% attenuation bound, not a model-quality threshold.
delta = 0.01
minimum_ratio = 1 / ((1 - delta) ** -2 - 1)
assert math.isclose(minimum_ratio, 49.2512562814, rel_tol=1e-10)
assert math.isclose(attenuation(minimum_ratio), 0.99)
inside = 0.001 / math.sqrt(0.001 ** 2 + 1e-6)
outside = 0.001 / (math.sqrt(0.001 ** 2) + 1e-6)
assert math.isclose(inside, 1 / math.sqrt(2))
assert math.isclose(outside, 1000 / 1001)
print("q/eps for 1% attenuation:", round(minimum_ratio, 6))

The five ratios produce 0.099504, 0.301511, 0.707107, 0.953463, and 0.995037. The assertions also cover zeros, large positive and negative values, an epsilon change, and rejection of empty inputs, NaN, and nonpositive epsilon. These assertions were executed locally. No GPU benchmark or model training was performed.

For the same input [0.001,0.001], changing epsilon from 1e-6 to 1e-4 changes each output from approximately 0.707107 to 0.099504. The code also checks a parenthesis mismatch: with x=0.001 and epsilon=1e-6, adding epsilon inside the square root yields about 0.707107, while adding it outside yields about 0.999001. Both implementations return finite values, yet their numerical behavior clearly differs. An acceptance rule that checks only finiteness would miss exactly this case.

What would a 1% attenuation limit require?

The ratio can be turned into an explicit condition. As a teaching example, suppose the additional attenuation before learned gain must not exceed 1%, meaning A≥0.99. From A=1/√(1+epsilon/q), the condition is epsilon/q≤1/0.99²−1. Equivalently, q/epsilon must be at least approximately 49.2513.

This is a bound on the RMS attenuation of one fixed input vector before gain. It does not bound model error to 1%, nor does it constrain changes in downstream accuracy, probabilities, or generated text to that percentage. Later layers transform the result; a local ratio cannot replace task acceptance.

With a known epsilon, this condition identifies inputs outside a chosen local tolerance. With a known q range to cover, it helps estimate epsilon relative to that range. For existing weights, a violation is something to record and explain before changing configuration. The 1% limit is an illustrative observation rule, not a universal requirement.

Preserve model configuration before choosing a new value

When migrating inference for existing weights, treat epsilon as part of the model configuration rather than an arbitrary deployment-side repair knob. Fix the inputs, weights, normalization placement, and parameters before comparing one change at a time. Changing precision, epsilon, and kernel simultaneously makes the source of a discrepancy difficult to isolate.

For a new training experiment, record the chosen epsilon and inspect the q distribution of the relevant activations. This reveals whether many examples occupy the region where q and epsilon are similar. That distribution must come from your own authorized measurements; the five teaching points in this article cannot stand in for model observations. Numerical stability and training or task results should inform the decision, not merely the impression that a number looks small.

Check framework defaults against the precise version in use. The referenced PyTorch documentation exposes epsilon as a configurable argument. The explicit 1e-6 used here is not a claim about every model or framework version. Record the resolved numerical value in the deployment configuration so that the word “default” cannot conceal a mismatch.

Separate error sources during a precision migration

Squaring, reduction, division by dimension, square root, and elementwise gain can each affect numerical results. The large-input example illustrates one issue: a finite mathematical result does not guarantee finite intermediates under a direct evaluation order. Passing this Python reference is not evidence that a low-precision path on the target device has been validated.

Keep a fixed higher-precision reference and change one condition at a time. Align epsilon first and compare the same inputs and gains. Then inspect reduction and output types, evaluation order, and whether an implementation is fused. Record maximum absolute and relative error for each change rather than leaving a description such as “close enough.”

Near-zero reference values make relative error sensitive, so relative tolerance alone is insufficient. One elementwise rule is |actual−reference|≤atol+rtol×|reference|, alongside a separate nonfinite-value check. Choose atol and rtol from the target precision and task requirements. This article does not invent a threshold that supposedly suits every deployment.

Leave an acceptance record that can be reviewed

A useful record includes vector dimension, normalization axes, input and gain types, resolved epsilon, reference implementation, error definition, and tested cases. Zero, small, ordinary, constant, and mixed-scale vectors reveal different problems. Covering those categories is useful without claiming that they exhaust the input domain.

If a discrepancy appears only for small inputs, inspect q/epsilon first. If it appears only at large magnitudes, examine intermediates. If particular channels differ, inspect gain and broadcasting. Narrowing the investigation from the observed pattern leaves a more credible explanation than repeatedly increasing epsilon until an error disappears.

A formula and scalar examples help explain why outputs differ. Task-level validation determines whether the discrepancy is acceptable. Keeping those questions separate turns epsilon from a copied decimal into a model setting whose rationale and effect can be reproduced.

References

Root Mean Square Layer Normalization, original 2019 paper; PyTorch 2.14 RMSNorm definition and parameters. The graphics and numbers are original derivations and local scalar calculations. The paper’s speed results were not reproduced, and model quality was not measured.