zero_grad(set_to_none=True) and clearing a gradient tensor to zero are not merely different memory operations. For PyTorch optimizers, grad=None skips that parameter’s step, whereas an existing zero tensor can still enter the update. Weight decay or historical momentum may then change the parameter.

No new gradient this batch, a derivative that happens to equal zero, and a frozen parameter are different claims. Two exact update paths below isolate their consequences and lead to practical checks for conditional branches and staged fine-tuning. 中文版.

Separate missing gradients from zero values

The PyTorch 2.8 zero_grad documentation distinguishes skipping a parameter with None from stepping it with a zero gradient. After set_to_none=True, parameters that receive no gradient retain grad=None. This discussion uses that built-in-optimizer contract, not a universal rule for every custom optimizer.

None is the absence of a gradient tensor. A zero tensor is an existing object with a shape and zero-valued elements. A parameter can participate in differentiable computation whose derivative cancels to zero. Conversely, an inactive parameter can still hold a tensor left from an earlier zeroing operation.

Record whether grad exists, its values or norm when present, and whether the parameter remains in an optimizer group. Mapping None to numeric zero for convenient logging destroys the distinction needed to explain these updates.

Decay can run without a new nonzero derivative

The AdamW definition separates weight decay from gradient moment estimation. Isolate decay in a teaching step with no historical moment contribution: θ=1, learning rate η=0.1 and decay λ=0.1. If a zero gradient enters the step, the decay term alone gives θ′=(1−ηλ)θ=0.99.

With grad=None and the stated skip behavior, the parameter does not execute that step and stays at one. This does not predict 0.99 for every AdamW zero-gradient update. Existing first and second moments and step counts also influence the complete algorithm.

The example establishes that zeroing does not mean doing nothing to a parameter. Decay belongs to the update procedure and does not require a nonzero current derivative. For a layer activated only in selected batches, missing versus zero gradients can change how often decay is applied.

Historical momentum can move a zero-gradient parameter

Consider the SGD momentum formula with an existing buffer, no decay, no dampening and no Nesterov: b′=μb+g and θ′=θ−ηb′. With θ=1, b=2, μ=0.9 and η=0.1, g=0 still produces b′=1.8 and θ′=0.82.

Skipping that parameter with None instead preserves θ=1 and b=2 in this teaching state. Its next activation resumes from a different buffer depending on whether previous updates ran. Looking only at the current derivative cannot explain the later trajectory.

A zero learning rate is not sufficient to freeze all optimizer state either. The zero-gradient momentum path can leave θ unchanged while reducing b from two to 1.8. Parameter immobility and state immobility need separate checks because they have different consequences when training resumes.

Nine checks compare both parameters and state

The two functions independently isolate decay and existing momentum. They are not full SGD or AdamW implementations: first-step initialization, bias correction, mixed precision and tensor backends are omitted. Their None branches preserve state according to the skip contract under discussion.

# Exact teaching transitions, not complete SGD/AdamW implementations.
from fractions import Fraction as F

def decay_only(theta, grad, lr=F(1,10), wd=F(1,10)):
    if grad is None:
        return theta
    if grad != 0:
        raise ValueError("this toy handles zero gradients only")
    return theta*(1-lr*wd)

def momentum_step(theta, buffer, grad, lr=F(1,10), mu=F(9,10)):
    if grad is None:
        return theta,buffer
    next_buffer=mu*buffer+grad
    return theta-lr*next_buffer,next_buffer

theta=F(1)
assert decay_only(theta,None)==1
assert decay_only(theta,F(0))==F(99,100)
assert decay_only(theta,F(0),wd=F(0))==1
assert momentum_step(theta,F(2),None)==(F(1),F(2))
assert momentum_step(theta,F(2),F(0))==(F(41,50),F(9,5))
assert momentum_step(theta,F(0),F(0))==(F(1),F(0))
assert momentum_step(theta,F(2),F(0),lr=F(0))==(F(1),F(9,5))
a,b=momentum_step(theta,F(2),F(0))
assert momentum_step(a,b,F(0))==(F(329,500),F(81,50))
assert decay_only(theta,None)!=decay_only(theta,F(0))
print("9 missing-versus-zero gradient checks passed")

All nine assertions passed locally, covering missing gradients, zero gradients, zero decay, no historical momentum, zero learning rate and two consecutive zero-gradient steps. The second momentum step gives θ=0.658 and b=1.62, showing how state changes persist. Exact fractions avoid roundoff ambiguity without measuring real training quality.

set_to_none=False does not promise a new zero tensor for every parameter

The versioned zero_grad implementation only enters the clearing path when a gradient already exists. A parameter that never received a gradient and currently has None should not be assumed to acquire an allocated zero tensor solely from set_to_none=False. The comparison here concerns the state actually observed by the optimizer, rather than a guess based on the option name.

For a conditional branch, first run a batch that activates the parameter and produces its gradient. Clear it to None in one path and zero in another. Then use an inactive batch and compare before and after step. This tests a parameter that previously had a gradient but receives none now. A never-activated new parameter may remain None in both paths and hide the difference.

Also exclude stale gradients from a previous batch. Deliberate accumulation requires preserving contributions across microbatches; ordinary updates require a clear initial state. The clearing boundary and whether a parameter participates in an update are related decisions, but one does not define the other.

The skip discussed here is per parameter. Other parameters can still update, while a global learning-rate scheduler or external counter advances separately. Preserving this parameter’s optimizer state is not the same as pausing the whole training loop.

Define what freezing is intended to preserve

Freezing may mean preventing new gradient recording, preventing parameter updates, or preserving historical optimizer state as well. Choose the intended contract before managing requires_grad, gradient clearing and optimizer groups. One switch cannot stand in for all three definitions.

After changing requires_grad, inspect what step will actually read if an old grad still exists and the parameter remains optimizer-managed. Whether to remove it from a group, retain momentum for resumption or reset historical state belongs to an explicit staged-training design rather than an improvised debugging assumption.

Restoring training also requires more than matching weights when update trajectories matter. Check optimizer state, the learning-rate schedule and the participating parameter set. The earlier AdamW versus L2 article explains update terms; this article asks whether those terms execute for a particular parameter.

Use one real step as the acceptance test

In the supported project environment, create identical model and optimizer-state copies. Confirm the chosen parameter already has history, construct None in one copy and zero in the other, execute one step, and compare parameter values plus relevant state. Loss alone is inadequate: update differences need not alter the forward loss calculated before that step.

Then validate the project’s actual foreach or fused path, AMP behavior, parameter sharing and conditional branches. For custom optimizers, read the implementation of the None case. Those backend tests were not run here, so nine arithmetic assertions do not certify a complete training stack.

A useful report retains whether the update path ran, θ before and after, state before and after, gradient existence, and configuration version. These facts explain more than the assumption that a zero gradient must leave a layer unchanged.

Do not compress different states into one zero

Choose set_to_none with update semantics and the actual implementation in mind, not just reduced memory writes. Decide whether inactive parameters should decay or advance state, then verify the optimizer’s contract. No gradient-clearing choice is universally correct independently of the intended training procedure.

If the examples and state comparison help explain why a zero-gradient parameter still moves, voluntary support through the section below is welcome. Preserving the distinction between missing and zero gradients makes staged training easier to reproduce.

Sources and backfill note

AdamW and zero_grad、SGD

Definitions were checked against versioned PyTorch 2.8 material available before the planned slot. The derivation, arithmetic code and two figures are original teaching material. The planned article date is October 19, 2026 at 23:30 Beijing time; actual publication is reported separately. No PyTorch training, distributed process or GPU benchmark ran.