AdamW weight decay is different from adding an L2 term to the objective. L2 adds lambda*w to the data gradient, allowing it to enter Adam’s first and second moments. AdamW separates parameter shrinkage from the adaptive gradient update. Matching learning rate and decay coefficients does not guarantee a matching step.

Zero data gradient makes the distinction easy to see. An exact scalar update below leads into the practical roles of learning-rate schedules, actual update counts, and parameter groups. 中文版.

Start with the ordinary SGD identity

For data loss L(w), adding lambda*||w||²/2 produces gradient g+lambda*w. Ordinary SGD without momentum gives w′=w−eta(g+lambda*w)=(1−eta*lambda)w−eta*g. Under this coefficient convention, it is equivalent to proportional shrinkage plus a data-gradient update.

Coefficient conventions matter too: writing lambda*||w||² without the factor of 1/2 gives a contribution of 2*lambda*w. Reusing the same coefficient would change the comparison. That identity relies on an update linear in the gradient. Adam constructs moment estimates from gradient history and scales coordinates adaptively. Putting lambda*w inside the gradient also changes those states, so the plain-SGD algebra cannot simply be transferred.

The Decoupled Weight Decay Regularization paper addresses this distinction. It was first submitted in 2017 and published at ICLR 2019. This is a mechanism explanation, not a new-release claim or a guarantee that the paper’s experimental outcome holds for every task.

What is actually decoupled?

Write gₜ for the gradient of the data objective being minimized. AdamW’s moments are formed from gₜ without mixing in lambda*w, while the parameter update also includes shrinkage −etaₜ*lambda*w. The PyTorch 2.14 algorithm reference explicitly separates decay from momentum and variance.

Decoupled does not mean independent of learning rate. In this implementation form, each shrink factor remains 1−etaₜ*lambda. Changing the learning-rate schedule changes the decay-only path even when the weight_decay configuration is unchanged.

An optimizer name is not a complete specification either. Implementations can offer decoupling options, and versions and settings matter. Compare the actual formula and configuration rather than assuming the class name alone establishes which update was used.

One zero-gradient step gives two different values

Choose a deliberately simplified scalar setting: w=1, data gradient g=0, eta=0.1, lambda=0.1, epsilon=0.001, and beta1=beta2=0. Removing historical moments makes the adaptive denominator |g|+epsilon. These are teaching conditions, not default settings or recommended training hyperparameters.

With L2 coupled into the gradient, the effective gradient is 0.1. The change is 0.1*0.1/(0.1+0.001)=10/101, yielding w′=91/101≈0.90099. AdamW has no data-gradient contribution here and instead gives (1−0.1*0.1)*1=0.99.

This is an algorithmic difference, not floating-point rounding. The counterexample disproves the claim that AdamW merely renames L2, but it does not choose the best lambda for a real model.

Eight exact checks reproduce the distinction

The code uses only Python’s standard-library Fraction. It implements the beta1=beta2=0 teaching formulas with strictly positive epsilon; no PyTorch optimizer, GPU, or fine-tuning run was executed.

# Scalar teaching update, beta1=beta2=0. No PyTorch/model run.
from fractions import Fraction as F

def adam_l2(w,g,eta,lam,eps):
    regularized = g+lam*w
    return w-eta*regularized/(abs(regularized)+eps)

def adamw(w,g,eta,lam,eps):
    return (1-eta*lam)*w-eta*g/(abs(g)+eps)

w,g,eta,lam,eps = F(1),F(0),F(1,10),F(1,10),F(1,1000)
a,b = adam_l2(w,g,eta,lam,eps),adamw(w,g,eta,lam,eps)
assert b == F(99,100)
assert a == F(91,101)
assert a != b
assert adam_l2(w,F(2,5),eta,0,eps)==adamw(w,F(2,5),eta,0,eps)
assert adamw(w,g,0,lam,eps)==w
assert adamw(-w,g,eta,lam,eps)==-F(99,100)
x=w
for _ in range(10):
    x=adamw(x,g,eta,lam,eps)
assert x==F(99,100)**10
assert adamw(w,g,2*eta,lam,eps)==F(98,100)
print("8 update checks passed; L2=91/101, AdamW=99/100")

Eight checks passed locally: both exact results, their inequality, agreement at zero decay, no change at zero learning rate, shrinkage of a negative parameter, ten-step decay, and the effect of doubling learning rate. The small functions are not presented as complete AdamW training implementations.

The same decay value accumulates differently over time

Without data updates, the decay-only path is wₜ=w₀∏ₛ(1−etaₛ*lambda). Constant learning rate reduces it to w₀(1−eta*lambda)ᵗ. Lambda is not a fixed end-of-run shrink percentage; learning rate and the number of actual updates enter the product.

Interpreting each factor as a nonnegative retained fraction requires 0≤etaₛ*lambda<1. Outside that interval, a factor may reverse the sign rather than just shrink the weight. This is not a convergence criterion.
At eta*lambda=0.01, one step retains 99%, ten retain about 90.44%, and one hundred about 36.60%. This demonstrates accumulated shrinkage, not proportional changes in validation quality. Real training additionally includes data gradients and moment state.

If an accumulation change alters optimizer.step count under the same training-token budget, compare the resulting decay schedule too. The earlier gradient accumulation article concerns objective normalization. Here the additional effect is shrinkage applied at actual optimizer updates.

A zero gradient is different from an absent gradient

In the example, g=0 means an existing zero-valued gradient, not .grad being None. The PyTorch zero_grad notes distinguish a zero-gradient step from skipping a parameter whose gradient is None.

An unchanged parameter therefore does not by itself show that weight decay is ineffective. Check whether the parameter belongs to the optimizer, whether it received a gradient, whether an update occurred, and which learning rate and decay apply to its group.

In general AdamW, zero current data gradient can still leave a nonzero adaptive update from historical moments. Zero gradient corresponds to pure shrinkage in this example because beta values are zero and history has been removed. Comparing a real optimizer requires its state as well as its current gradient.

Parameter groups decide where decay applies

Different groups can carry different learning rates and decay values. One global weight_decay number does not prove all trainable parameters share that setting. Inspect the final parameter lists and group configurations, rather than only a default value in a configuration file.

Choose exclusions from model and training requirements instead of treating an online rule as universal. Also check for omitted parameters, duplicate inclusion, and accidental changes to group correspondence when resuming a run. These are configuration questions separate from the mathematical update distinction.

If the objective already includes explicit L2 while AdamW decay is enabled, both coupled regularization and decoupled shrinkage are present. That may be intentional, but it should not be described as applying weight decay only once. Record both contributions.

Compare one update before comparing entire recipes

Fix a small parameter, gradient, learning rate, and optimizer state. Compare expected and actual changes before restoring momentum, mixed precision, clipping, and real data. Check clipping order as well: explicit L2 mixed into a gradient before clipping also enters that clipping operation. AdamW’s separate shrinkage does not thereby become the same formula. A final loss curve cannot uniquely diagnose which of those conditions differed.

A reproducible resume also needs optimizer state and scheduler progress. Loading weights while rebuilding zero moments is a different starting point from restoring the full optimizer. A seed is not the first explanation to reach for when those states do not match.

No universal best lambda is supplied here. The useful output is an inspectable update contract: identify the data gradient, the quantities entering moments, the per-group shrinkage operation, and the number of updates. If the counterexample helps untangle AdamW and L2 in a training recipe, voluntary support through the section below is welcome.

References and validation scope

Checked October 17, 2026 against the original AdamW paper, PyTorch 2.14 AdamW reference, and zero_grad notes. Both diagrams, the scalar construction, and code are original. Local validation excludes actual optimizer execution, model performance, and distributed training.