If a custom attention layer still changes its output after eval(), inspect the dropout_p passed to scaled_dot_product_attention. The function applies dropout according to that explicit argument. Calling eval() on the enclosing model does not rewrite a hard-coded nonzero probability, and neither no_grad nor inference_mode supplies that missing argument change.

The useful boundary is the actual call, rather than another outer inference context. An exact two-weight enumeration below explains how dropout can preserve the mean while changing every individual output in the example. 中文版.

Three controls with separate responsibilities

The PyTorch 2.14 SDPA reference warns that the function applies dropout according to dropout_p and requires 0.0 during evaluation. In a custom module, self.p if self.training else 0.0 is the usual bridge from module state to the function argument.

The autograd notes distinguish evaluation mode from no-grad and inference mode. Evaluation changes behavior implemented by training-aware modules. The other modes control autograd tracking and related overhead. Whether a forward pass records gradients is a different question from whether it draws a random mask.

Check both the owning module’s intended mode and the probability reaching SDPA. Printing model.training alone does not prove that every custom attention path passed zero. The mode matters only where the implementation connects it to the operation.

Why experience with nn.Dropout can mislead

When nn.Dropout is placed in evaluation mode, it acts as an identity function, as its module documentation describes. A direct functional call inside forward follows its own argument contract. The convenience of a module does not imply that every dropout-related function inspects its parent’s state.

This is easy to miss when replacing handwritten attention with SDPA: the configured probability survives, but the training-state branch disappears. The optimized path then retains a random operation during evaluation. The repair belongs at the call boundary, rather than in checkpoint weights or an unconditional change to the training configuration.

Keep the configured p if training should retain regularization, and select the effective probability at invocation. Permanently setting the configuration to zero can stabilize evaluation while silently changing the intended behavior of later training.

Equal expectation does not mean equal output

Fix post-softmax weights a₁=a₂=1/2 and scalar values v₁=2, v₂=4. Without dropout, the weighted output is y=Σaᵢvᵢ=3. Queries, keys, and values are fixed; no text-generation sampling or training is involved.

Let p=1/2 be the drop probability, with independent keep indicators mᵢ whose probability of being one is 1-p. Inverted dropout uses ãᵢ=aᵢmᵢ/(1-p). For 0≤p<1, E[ãᵢ]=aᵢ, so a linear weighted sum of fixed values satisfies E[ỹ]=y. This expression does not handle the p=1 all-dropped boundary; its denominator cannot be set to zero.

The four masks 00,10,01,11 are equally likely here and produce 0,2,4,6 respectively. Their average is 3, yet none of the individual values equals 3. Independence also gives Var(ỹ)=p/(1-p)·Σ(aᵢvᵢ)², which becomes 1²+2²=5 here, agreeing with the enumeration. Preserving expectation therefore neither makes each forward equal to its no-dropout counterpart nor makes repeated forwards equal to each other.

Enumerate the possibilities instead of sampling

This standard-library Python example uses exact fractions and enumerates all four masks. It validates the mathematical construction. PyTorch is not installed in the current environment, so no SDPA call, neural model, or GPU kernel was executed; the API contract comes from the official documentation. The final effective_p function demonstrates a Python policy, rather than inspecting a real PyTorch module.

# Exact arithmetic for a teaching example; not a PyTorch execution.
from fractions import Fraction as F
from itertools import product

weights, values = [F(1,2),F(1,2)], [F(2),F(4)]
p = F(1,2)
base = sum(a*v for a,v in zip(weights,values))
outputs, sums = [], []
for keep in product([0,1],repeat=2):
    dropped = [a*m/(1-p) for a,m in zip(weights,keep)]
    outputs.append(sum(a*v for a,v in zip(dropped,values)))
    sums.append(sum(dropped))
mean = sum(outputs)/len(outputs)  # all four masks equally likely here
variance = sum((x-mean)**2 for x in outputs)/len(outputs)
assert base == 3
assert sorted(outputs) == [0,2,4,6]
assert mean == base
assert variance == 5
assert sorted(sums) == [0,1,1,2]
assert sum(sums)/len(sums) == 1
# Explicit evaluation policy: p must become zero at the call boundary.
def effective_p(training, configured):
    return configured if training else F(0)
assert effective_p(False,p)==0
assert effective_p(True,p)==p
print("8 exact checks passed; mean=3, variance=5")

Eight assertions passed locally: baseline output, possible outputs, mean, variance, possible weight sums, their mean, and both effective-probability branches. Equal probabilities justify the simple average only for the p=1/2 construction. For other probabilities, masks require their correct probability weights.

The weights need not sum to one afterward

A softmax row sums to one before dropout. That does not imply the row still sums to one afterward. The example’s sums are 0,1,1,2; only their expectation is one. Multiplication by 1/(1-p) compensates expectation rather than renormalizing each realized row.

Dividing surviving weights by their row sum to “repair the probability distribution” changes the operation. An all-zero row adds a division-by-zero case. Keep properties attached to the stage where they hold; otherwise a correct implementation can appear wrong because it is checked against an invariant from an earlier stage.

The equality E[ỹ]=y also concerns this fixed-value linear combination only. Once a nonlinear function, normalization, or another random operation follows, expectation cannot generally be moved inside that function. It does not establish that a network’s average stochastic prediction equals its evaluation prediction.

A reset seed can conceal an active dropout path

Resetting the same random state before every invocation can reproduce a mask even when dropout was accidentally left active. Identical observed outputs then demonstrate a repeated random process, not an absence of dropout.

Conversely, seeding once does not require consecutive calls to use the same mask, because calls consume random state. Equality alone is a weak test in another way: particular values or symmetries can let different masks produce the same output. The test’s sensitivity depends on the construction as well as the implementation.

Check directly that evaluation calls receive 0.0, then compare expected outputs on fixed inputs. Test the training path separately to ensure its nonzero configured probability remains intact. Argument flow and an explanatory counterexample are more informative than repeatedly changing a seed until an output looks stable.

After disabling dropout, isolate other differences

Compare the same Q,K,V and mask first, excluding preprocessing changes. Then inspect the effective probability and dtype before comparing local outputs under the same backend. Starting with generated text combines sampling choices and accumulated numerical differences with the local attention question.

Disabling dropout does not promise bitwise equality across devices, backends, and precisions. The SDPA reference notes numerical differences among implementations. Choose tolerances from actual numerical conditions and task needs, and distinguish small implementation-dependent differences from the random operation discussed here.

The earlier KV cache causal-mask article concerns which positions may participate. This article concerns whether permitted weights are then randomly removed. Both defects can coexist, so resolving one is not evidence that the other has been validated.

Make the inference contract inspectable

The smallest useful change is often to retain the training configuration while selecting zero explicitly for evaluation, with an inspectable effective value at the call. Regression coverage should exercise both modes so a later refactor cannot quietly drop the self.training branch.

Record framework version, input shape, dtype, effective dropout probability, mask settings, and backend. Compare those conditions before interpreting a numerical discrepancy. This article supplies no unexecuted model-performance or stability benchmark.

If the derivation helps distinguish “no gradient tracking” from “no random dropping,” or shortens an inference discrepancy investigation, voluntary support through the section below is welcome.

References

Checked October 16, 2026 against the PyTorch 2.14 SDPA reference, Dropout module reference, and autograd notes. Both diagrams and the exact enumeration are original; local validation is limited to the standard-library mathematical example.