If an inference call already uses no_grad but one sample changes when its batch companions change, check whether BatchNorm still uses current-batch statistics. Disabling gradient recording does not select evaluation behavior. Conversely, eval does not guarantee fixed statistics under every configuration.

This example separates the forward calculation from reverse-mode recording, constructs a sign change for the same input, and follows the running statistics that deployment must preserve. 中文版.

Separate three controls before debugging

The BatchNorm1d definition distinguishes current-batch normalization during default training from saved running statistics during evaluation. With track_running_stats=False, batch statistics are used during evaluation too. no_grad controls gradient recording; it does not switch a module into evaluation mode.

Describe a forward call using three separate facts: the module’s training state, whether running statistics are tracked, and the gradient mode. Freezing learnable parameters answers another question again. Whether parameters receive gradients is distinct from which mean and variance determine an output.

The arithmetic below has one channel and affine coefficients γ=1 and β=0. Real BatchNorm1d can aggregate over batch and length dimensions. Per-channel normalization of a two-dimensional input should not be mistaken for independent normalization of every sample.

How the same input changes sign

Batch A=[0,2] has mean one and variance one when the sum of squared deviations is divided by the number of elements. Its value two becomes 1/√(1+ε), approximately positive one. Batch B=[2,4] has mean three and the same variance. The same value two now becomes −1/√(1+ε).

No dropout or floating-point discrepancy is needed. The value is unchanged; the companions contributing to its statistics have changed. As long as the forward calculation uses those statistics, disabling gradient recording cannot mathematically remove their influence.

This does not predict a visible sign change for every full network. Affine transforms, later layers and input distributions influence the observed effect. It supplies a minimal mechanism from which to design a controlled project test, rather than an empirical claim about a deployed model.

With fixed saved statistics, the normalized scalar is γ(x−μ_run)/√(v_run+ε)+β. Holding x, γ, β, μ_run and v_run fixed removes batch companions from this particular formula. It does not remove other cross-sample dependencies elsewhere in a network.

The running variance is not a copy of the forward variance

PyTorch 2.8 specifies a biased variance estimate for training-time normalization and an unbiased estimate for updating the running variance. In the two-element example those are one and two respectively. Starting from running_var=1 with momentum=0.1 gives 0.9×1+0.1×2=1.1.

The mean also blends old and new statistics. An old mean zero and current mean one give an updated mean 0.1. This momentum is the mixing coefficient for a new statistic, not optimizer momentum. Remembering only that a moving average exists, without identifying the variance estimator, can make a hand calculation disagree for a valid reason.

A training output need not equal the immediately following evaluation output. One uses the current batch, while the other can use a blend of historical observations. Even identical data therefore does not imply identical formulas. Record both sets of statistics in local diagnostics before increasing numerical tolerances.

Eight arithmetic checks reproduce the distinction

The code explicitly computes a mean, two variance estimators and one running-statistic update using the standard library. It neither implements the complete module state machine nor validates a PyTorch device backend.

# One-channel teaching arithmetic, not a PyTorch module execution.
from math import sqrt, isclose

def stats(xs):
    mean = sum(xs)/len(xs)
    biased = sum((x-mean)**2 for x in xs)/len(xs)
    unbiased = biased*len(xs)/(len(xs)-1)
    return mean, biased, unbiased

def normalize(x, mean, var, eps=1e-5):
    return (x-mean)/sqrt(var+eps)

m1,v1,u1 = stats([0.,2.])
m2,v2,u2 = stats([2.,4.])
assert (m1,v1,u1)==(1.,1.,2.)
assert (m2,v2,u2)==(3.,1.,2.)
assert normalize(2.,m1,v1)>0
assert normalize(2.,m2,v2)<0
running_mean,running_var,momentum = 0.,1.,0.1
running_mean=(1-momentum)*running_mean+momentum*m1
running_var=(1-momentum)*running_var+momentum*u1
assert isclose(running_mean,0.1)
assert isclose(running_var,1.1)
a=normalize(2.,running_mean,running_var)
b=normalize(2.,running_mean,running_var)
assert a==b
assert not isclose(a,normalize(2.,m1,v1))
print("8 BatchNorm arithmetic checks passed")

All eight assertions passed locally. Four verify batch statistics and the sign change; two verify the updated running values; the last two compare a fixed-statistic result with itself and with the current-batch result. Repeating one formula is a control in this example, not proof that every model is deterministic.

Make batch perturbation a one-variable test

Keep the target sample and its position fixed, placing it with two different sets of companions in valid batches. Compare its outputs under the same module mode, then identify the statistics each BatchNorm layer actually uses. Both batches need enough valid statistical elements; an invalid single-element training input is not a useful functional control.

Next, for the default tracking configuration, switch to evaluation and rerun from the same saved state. If batch dependence disappears, the observation points toward the batch-statistic path. If it remains, inspect track_running_stats, other cross-sample operations, preprocessing and randomness. One comparison narrows the investigation; it does not certify the whole pipeline.

Control dropout so that a randomness difference does not obscure the intended statistics comparison. If position, length or masks change simultaneously, the experiment no longer isolates batch companions. The earlier padding-participation investigation uses the same principle: first establish which inputs truly stay fixed.

Preserve statistical state as well as weights

If deployment depends on running_mean and running_var, saving only γ and β cannot reconstruct the same forward function. Running statistics are not optimizer parameters, yet change evaluation outputs. Include the required buffers in restoration checks, and record the module mode, tracking configuration and preprocessing version.

Likewise, do not accidentally send validation data through a training path that updates statistics when fixed evaluation behavior is intended. No backward call is needed for a state update to affect later outputs. Re-estimating statistics on new data may be an explicit adaptation procedure, but should be evaluated as such rather than occurring accidentally.

A small fixed set of inputs and expected outputs can check a full save-and-load cycle. A separate companion-changing test checks the intended batch independence. The first tests restoration completeness; the second tests a forward-behavior requirement. Together they provide more evidence than successful loading and a forward call alone.

Identify the statistic source before tuning parameters

For batch-dependent inference changes, check module mode, track_running_stats, actual means and variances, and restored state before changing ε or retraining. Those changes alter behavior and cannot replace identifying which forward path was used.

If the counterexample and state checks help explain different outputs for the same input, voluntary support through the section below is welcome. The useful result to retain is a test that reproduces and localizes the discrepancy, rather than a switch combination that happens to stabilize one output.

Sources and backfill note

BatchNorm1d、no_grad

The definitions were checked against PyTorch 2.8 material available before the planned slot. Derivations, counterexamples and figures are original teaching material. The article keeps October 19, 2026 at 09:00 Beijing time as its planned date and was backfilled that evening. Standard-library code ran locally; no PyTorch model or device-performance benchmark ran.