Dropout preserves an activation’s mean during training, but a single training forward pass need not equal an evaluation pass. Even the average output of the whole network need not equal its output with dropout disabled. The common mistake is extending an expectation identity for one layer to every computation downstream.

For the question “Why divide dropout outputs by 1−p, and should evaluation rescale them again?”, a two-coordinate example supplies a derivation, a nonlinear counterexample, and exact executable checks. 中文版.

What the scaling compensates for

The PyTorch 2.8 Dropout definition zeros elements with probability p during training and scales retained values by 1/(1−p). In evaluation this module is an identity. Our derivation assumes 0≤p<1 and treats input x as fixed. A mask m is one with probability q=1−p and zero otherwise.

Writing y=mx/q gives E[y]=E[m]x/q=qx/q=x. This expectation is over mask randomness alone. The compensation concerns the single activation averaged over masks. It does not reconstruct a particular activation that was dropped, and does not require exactly a fraction q of a batch to survive. A small tensor can have a retained count far from its expectation without contradicting this calculation.

Since m²=m, E[y²]=x²/q and Var(y)=x²/q−x²=p x²/(1−p). Preserving the mean therefore does not preserve the variance. As p approaches one, increasingly rare surviving values become increasingly large. An account that mentions only an unchanged mean misses this practical distinction. The fraction is undefined at p=1, so this derivation does not describe that endpoint’s implementation.

Four masks, none equal to the evaluation output

Fix x=[2,4] and p=0.5, with independent masks for the two elements. The four equally likely outputs are [0,0], [0,8], [4,0], and [4,8], each with probability one quarter under the stated independent-mask model. Their coordinatewise mean is exactly [2,4]. Yet none of the four outputs is [2,4]. An expected value can be preserved even when no individual outcome equals it.

Consequently, equality between one training output and one evaluation output is an unsuitable acceptance test for dropout. Nor does reproducing the same mask with a fixed random seed establish that the random layer is disabled. Reproducing a random computation and removing that computation address different requirements: one reproduces an experiment, while the other changes the forward function.

This example enumerates every mask, so its mean is the exact expectation rather than a lucky finite sample. For n independent binary masks there are 2ⁿ combinations. Enumeration is therefore a useful mechanism check at small scale, not an efficient implementation for realistic tensors. Larger experiments need statistical tests chosen for their purpose, with sampling error kept distinct from a semantic error.

A ReLU changes the expectation comparison

Consider only the first coordinate. After training dropout it is zero or four with equal probability. Follow it with h(z)=max(0,z−3). These possibilities become zero and one, whose average is 0.5. Evaluation instead supplies the fixed input two to the head, yielding max(0,2−3)=0.

Thus E[h(y)]=0.5, whereas h(E[y])=0. The nonlinear threshold alone creates the discrepancy; neither floating-point error nor gradients nor a large model is required. A mean identity for an upstream activation is insufficient to require matching batchwise training and evaluation losses. Intervening nonlinear operations and the loss itself can invalidate the proposed equivalence.

A fixed affine head h(z)=az+b provides a useful control because E[h(y)]=h(E[y]) does hold there. Identify the downstream function before moving an expectation through it. A linear final layer does not rescue the argument if nonlinear operations sit between dropout and that layer. The relevant object is the whole intervening path, not just its last operation.

Exact rational arithmetic isolates the mechanism

The following standard-library program enumerates four masks and uses Fraction for exact arithmetic. Eight assertions check the mean, the complete outcome set, disagreement of every individual outcome with the input, the variance, both nonlinear-head paths and coordinate covariance. It does not run a PyTorch model or benchmark a device.

# Exact finite-mask arithmetic, not a PyTorch runtime test.
from fractions import Fraction as F
from itertools import product

x = (F(2), F(4))
p = F(1, 2)
q = 1 - p

def output(mask):
    return tuple(v * m / q for v, m in zip(x, mask))

ys = [output(m) for m in product((0, 1), repeat=2)]
means = tuple(sum(y[i] for y in ys) / len(ys) for i in range(2))
assert means == x
assert ys == [(0, 0), (0, 8), (4, 0), (4, 8)]
assert all(y != x for y in ys)
var0 = sum((y[0] - x[0]) ** 2 for y in ys) / len(ys)
assert var0 == p / q * x[0] ** 2 == 4
relu = lambda t: max(F(0), t)
random_head = sum(relu(y[0] - 3) for y in ys) / len(ys)
fixed_head = relu(x[0] - 3)
assert random_head == F(1, 2)
assert fixed_head == 0
assert random_head != fixed_head
cov = sum((y[0]-2)*(y[1]-4) for y in ys) / len(ys)
assert cov == 0
print('8 exact dropout arithmetic checks passed')

The zero covariance follows from the explicitly independent masks. It should not be generalized to dropout variants that share a mask over a channel. Inputs are fixed in this calculation too. With random inputs, first compute the conditional expectation for a given input and then account for the input distribution. Stating those assumptions prevents more confusion than merely increasing a sampling loop’s iteration count.

The checks distinguish two useful failure classes. A wrong retained-value scaling changes the expected activation even before the head. A test that demands equal nonlinear-head outputs can fail despite correct scaling. Keeping both checks small makes it possible to tell an implementation discrepancy from a mistaken requirement, rather than treating every numerical mismatch as a framework defect.

Evaluation mode and functional arguments are separate paths

The Module.eval definition is equivalent to train(False), selecting evaluation behavior for affected modules. That module path applies to ordinary nn.Dropout. Do not multiply its evaluation output by 1−p again: that adds an extra shrinkage absent from the intended calculation.

functional.dropout, however, accepts its own training argument, whose default is True. A forward implementation that calls it without passing training=self.training does not have that function argument rewritten by the parent module’s eval call. Follow the actual call path when inspecting the switch. Finding eval at the entry point is insufficient evidence about a separately supplied function argument.

Another valid requirement is deliberately retaining random forwards and sampling repeatedly. In that case, specify which layers should stay random and which must keep evaluation behavior. Switching the entire network back to training may also change layers such as BatchNorm. The earlier BatchNorm statistic-source investigation explains why those state choices need separate treatment. Repeated random outputs alone do not establish that an uncertainty estimate is calibrated.

Write acceptance criteria for the intended behavior

For fixed evaluation behavior, record the model mode and functional-call arguments, repeat a forward on the same input, state and preprocessing, then localize any change. Stable results support that controlled comparison; they do not certify determinism for every device, operation and input. Other randomness and nondeterministic operations require their own investigation.

For a scaling-mechanism test, use small inputs, known masks and an analytic expectation, keeping the scope at the layer boundary as above. Do not require a nonlinear network to satisfy an expectation identity that is mathematically false. A failing assertion becomes informative only when its expected behavior was valid in the first place.

When moving from this arithmetic to a project, preserve two separate records: which identity the test checks, and which runtime configuration was actually executed. This program verifies the former with exact values. A framework integration test would additionally need the concrete module or functional call, its mode and version, and the observed outputs. Those observations should be recorded when the integration test runs rather than inferred from this teaching model.

If these derivations and counterexamples help you write a more accurate dropout test, voluntary support through the section below is welcome. The reusable result is a method for separating a layer’s expectation, an individual outcome and whole-network behavior.

Sources and backfill note

Dropout · functional.dropout · Module.eval

Checked against versioned PyTorch 2.8 definitions available before the planned October 20, 2026 09:00 Beijing slot. This is a backfilled article retaining that planned date; the actual publication time is recorded separately. Diagrams and derivations are original. The standard-library arithmetic was executed; a PyTorch integration test was not.