Gradient accumulation can combine several microbatches into one update, but averaging their mean losses and dividing by the accumulation count is not universally equivalent to one larger batch. For a token-mean objective, unequal counts of contributing tokens change the weight assigned to each token.

Choose the objective’s denominator before choosing backward-pass scaling. A one-parameter exact example below compares the loss, gradient, and a single update, rather than trusting a logged average. 中文版.

Decide what should receive equal weight

Suppose an accumulation window contains K microbatches. Microbatch i has nᵢ valid tokens and summed token loss Sᵢ. Giving every valid token equal weight defines L=(ΣSᵢ)/(Σnᵢ). If the interface returns Lᵢ=Sᵢ/nᵢ, recover that objective with L=Σ(nᵢ/N)Lᵢ, where N=Σnᵢ.

Using ΣLᵢ/K gives each microbatch equal weight instead. Equal nᵢ is a special case where the weights agree. Coincidentally obtaining equal numbers on a particular input does not establish correctness for arbitrary length distributions.

Token weighting is itself a task choice, not the only valid objective. Some training intentionally weights examples equally despite different lengths. The question is whether the implementation matches the declared objective, rather than whether every mean should be replaced by a sum.

One scalar parameter reveals the changed gradient

Let the model predict the same scalar w for every valid item, with loss ℓ=(w-y)²/2. Batch A has one target 0; batch B has three targets 2. At w=0, their summed gradients are 0 and -6. The whole-window mean gradient is -6/4=-1.5.

Averaging per-batch mean gradients instead gives(0+(-2))/2=-1. The full-window loss here is 1.5, whereas the average of batch means is 1, so the logged value and gradient can both change. Both methods use identical data and parameters and update only once, yet the update magnitude differs. With learning rate 0.1, the valid-item mean moves w from 0 to 0.15, while the mean-of-means moves it to 0.1.

Squared loss makes the example hand-checkable. The argument uses linearity of summation and differentiation with a constant denominator, not a special language-model property. A sum of token cross-entropies similarly needs normalization that matches its intended objective.

Eight checks compare against the full window

Python’s standard-library Fraction keeps this quadratic construction exact. The code compares the full-window gradient, count-weighted microbatch gradients, and a central difference, with equal-count, empty-microbatch, and empty-window cases. No PyTorch, Transformers, or real-model training was executed.

# Scalar squared-loss teaching example; no training framework is run.
from fractions import Fraction as F

groups = [[F(0)], [F(2),F(2),F(2)]]
w = F(0)
def loss_sum(w, ys):
    return sum((w-y)**2/2 for y in ys)
def grad_sum(w, ys):
    return sum(w-y for y in ys)
def token_loss(w, batches):
    n = sum(len(ys) for ys in batches)
    if n==0:
        raise ValueError("no valid items")
    return sum(loss_sum(w,ys) for ys in batches)/n

def token_grad(w, batches):
    n = sum(len(ys) for ys in batches)
    if n==0:
        raise ValueError("no valid items")
    return sum(grad_sum(w,ys) for ys in batches)/n

n = sum(map(len,groups))
mean_of_means = sum(grad_sum(w,ys)/len(ys) for ys in groups)/len(groups)
weighted = sum(F(len(ys),n)*(grad_sum(w,ys)/len(ys)) for ys in groups)
global_g = token_grad(w,groups)
h = F(1,100)
finite_diff = (token_loss(w+h,groups)-token_loss(w-h,groups))/(2*h)
assert global_g == F(-3,2)
assert mean_of_means == -1
assert weighted == global_g
assert finite_diff == global_g and token_loss(w,groups) == F(3,2)
assert token_grad(w,[[F(0)],[F(2)]]) == -1  # equal counts
assert token_grad(w,[[],*groups]) == global_g
assert w-F(1,10)*global_g == F(3,20)
try:
    token_grad(w,[[],[]])
except ValueError:
    pass
else:
    raise AssertionError("empty window accepted")
print("8 gradient checks passed; global=-3/2, mean-of-means=-1")

All eight checks passed locally. The central difference is exact for this quadratic objective, not a claim that arbitrary finite-difference steps exactly differentiate general models. Weighted matches global_g for the specified fixed-window objective; mean_of_means exposes a different weighting choice.

Valid tokens are not simply attention-mask entries

Causal language-model labels are typically shifted into prediction alignment, and padding, prompt tokens, or other positions may be excluded. Count the target items actually participating in the loss rather than substituting input-token count, sequence length, or the original attention-mask sum.

The PyTorch 2.14 cross-entropy reference defines ignore_index and reduction behavior. This derivation assumes unweighted valid items. With class weights, example weights, or another objective, the denominator follows that loss definition rather than an unconditional count of nonignored positions.

The Hugging Face explanation published October 16, 2024 discusses the distinction between summed token losses divided by total valid tokens and averaging batch means. It is historical explanatory material, not evidence that every current Trainer has the same bug, and not a recommendation to install an old development branch.

Know the window denominator before backward scaling

For one device and a fixed window without additional automatic scaling, backpropagating each Sᵢ/N accumulates the intended gradient before one update. If only a correctly defined item mean is available, weight it by nᵢ/N. This requires knowing the window’s N and checking that no internal layer rescales it again. N counts labels participating in the objective and must not depend on w here. Learned weighting introduces additional differentiation dependencies.

When a framework already handles accumulation or valid-item normalization, another external division can shrink the gradient twice. Inspect the loss definition, returned reduction, backward path, and optimizer-step timing rather than inferring behavior from a configuration name.

An alternative accumulates unnormalized gradients and divides them by N before updating. That requires understanding the order of gradient scaling, clipping, and numerical-range management. The objective equation is useful, but it is not a universally validated patch to insert into an existing trainer.

Partial windows, empty labels, and multiple devices

The final data window may contain two microsteps while the configured maximum is four. Dividing by four without framework compensation changes update scale again. The items actually included in the window matter more directly than its configured maximum length.

A microbatch with no valid labels contributes zero summed loss and zero valid items; its mean cannot be formed by dividing by zero. An entirely empty window has no defined denominator and should be skipped or reported according to the training design, rather than silently recorded as a normal zero-loss update.

Multiple devices introduce global item counts and communication semantics: are gradients summed or averaged across ranks? A single-device expression is not automatically a DDP scaling rule. Establish a separate reference for the actual framework’s communication and normalization behavior. This example does not validate a distributed implementation.

Matching normalization is not complete training equivalence

Comparison with one large batch also assumes a decomposable forward/objective and unchanged parameters throughout the window. Calling optimizer.step after every microbatch creates a different update schedule from the one discussed here.

Dropout, batch-statistics-dependent layers, clipping timing, mixed precision, and floating-point summation order can affect the comparison. Isolate normalization with a small deterministic objective first, then restore other settings one at a time. A broad statement that accumulation must reproduce any training run omits these conditions.

The earlier perplexity article also aggregates valid tokens. Training adds a consequence: denominator changes enter backpropagation and alter parameter updates. Correcting a displayed loss does not by itself correct the gradient used by the optimizer.

Record enough to reproduce one update

Retain per-microbatch valid counts, summed losses, the window count, actual update count, and gradient comparisons for selected parameters. Include deliberately unequal lengths and an equal-length control. Passing only the equal-length case can conceal the weighting error.

Start with one window, identical initialization, and the same optimizer conditions rather than comparing final training metrics immediately. Once that update agrees, validate the actual data and framework path. Differences then have narrower explanations: objective weighting, forward randomness, or update timing, rather than an undifferentiated label of unstable training.

If the derivation and exact example help uncover a gradient mismatch hidden behind batch-mean losses, voluntary support through the section below is welcome.

References and validation scope

Checked October 17, 2026 against the Hugging Face historical explanation and PyTorch 2.14 cross-entropy definition. Both diagrams, the squared-loss construction, and code are original teaching material. Local validation does not include model fine-tuning or distributed execution.