Setting padding_idx on an Embedding layer does not automatically tell a later attention layer to ignore padding positions. Even an exactly zero pad vector can participate in the softmax denominator and change the weights assigned to real tokens. A two-value counterexample makes this visible: with both scores zero and values two and zero, unmasked output is one; excluding the pad gives two.

This walkthrough separates embedding lookup, attention masking, pooling and loss participation. Twelve exact-arithmetic checks reproduce the effect of added padding and provide a basis for tests in a real model. The code is a standard-library mathematical example; PyTorch and fused attention kernels were not run. 中文版.

padding_idx describes embedding lookup

The PyTorch Embedding documentation says that the padding_idx entry does not contribute lookup gradients. Its row defaults to zeros in a newly constructed module and may be changed to another padding vector. This is an embedding-layer rule, not a universal instruction that masks every subsequent operation.

Two independent questions follow: what vector does lookup return, and which keys may participate in attention? Combining them leads to the mistaken conclusion that a zero pad row removes the need for an attention mask. A later module that receives neither lengths nor valid-position information cannot automatically infer your padding convention.

Real networks can also add positional representations, biased projections or other transformations. A zero embedding need not remain zero afterward. Even without those additional effects, the normalization counterexample below already applies.

A zero value remains in the denominator

Let one query have two key scores s₁=s₂=0. Softmax assigns both weights exp(0)/(exp(0)+exp(0))=1/2. With v₁=2 and v₂=0, the output is (1/2)×2+(1/2)×0=1.

The second value contributes nothing to the numerator, yet reduces the first value’s weight from one to one-half. Explicitly excluding the second key leaves only the first participating in normalization, restoring output two. A mask changes membership of the normalization set; zeroing a member’s value does not do that.

Zero scores can be constructed using a zero query. Keys and values are independently specified teaching quantities here. They are not measured hidden states from a particular Transformer, and the example does not assume that real attention scores are normally all zero.

Padding length exposes the mistake

Append m zero-valued pads after the same valid value two, retaining zero scores everywhere. Without masking, each weight is 1/(m+1), so the result becomes 2/(m+1). Adding pads shrinks the result even though the valid content never changed.

This suggests a useful regression test: fix valid tokens and their positional meaning, vary only padding count, and check expected agreement at valid positions. A change should prompt examination of masks and reductions. The observation alone does not establish a defect in a particular kernel.

Control dropout and other randomness when making that comparison, and allow appropriate floating-point tolerances for the actual implementation. Left padding that also changes position IDs is not a one-variable test of padding count. Align positional meaning before interpreting its differences.

Twelve checks reproduce the denominator effect

The code uses the stated condition that every allowed score is zero, directly computing a uniform weighted average. It avoids exponential approximations and is not a general softmax implementation. Empty and fully masked sets are explicitly rejected by this application-level contract.

# Exact attention toy: all logits are zero, so allowed weights are uniform.
from fractions import Fraction as F

def zero_score_attention(values, allowed):
    if len(values) != len(allowed):
        raise ValueError("shape mismatch")
    kept = [F(v) for v,ok in zip(values,allowed) if ok]
    if not kept:
        raise ValueError("no valid key")
    return sum(kept,F(0))/len(kept)

for pads in [0,1,2,5]:
    values = [2] + [0]*pads
    assert zero_score_attention(values,[True]*len(values)) == F(2,1+pads)
    assert zero_score_attention(values,[True]+[False]*pads) == 2
assert zero_score_attention([2,6,0],[True,True,False]) == 4
for values,mask in [([],[]), ([0],[False]), ([2],[True,False])]:
    try:
        zero_score_attention(values,mask)
    except ValueError:
        pass
    else:
        raise AssertionError("expected ValueError")
print("12 padding-attention checks passed")

Twelve checks passed locally: four padding counts each check an unmasked and correctly masked result; one checks averaging two valid values; three check rejection of empty input, no valid keys and mismatched lengths. Raising an exception is the teaching function’s choice. It does not establish identical fully masked-row behavior across frameworks.

Boolean mask meaning depends on the API

In scaled_dot_product_attention, True in a boolean attn_mask means participation. A floating-point mask is added to attention scores. A floating zero therefore does not exclude a position: it merely adds zero. Construct visibility according to the target API rather than guessing from a variable name.

MultiheadAttention key_padding_mask uses the other boolean convention: True means the corresponding key is ignored. For that key_padding_mask argument specifically, a tensor where True means valid cannot be reused unchanged as one where True means padding. This comparison does not assign one universal meaning to every mask argument. When migrating interfaces, checking polarity matters more than merely checking dtype.

Write down the meaning of every dimension: batch, head, query position and key position. A broadcastable shape proves only that the operation can execute. It does not prove that a sample’s valid keys are applied to its own queries. Inspect a short visibility matrix for two unequal-length examples before expanding to full batches.

Masked keys do not imply zero pad-query outputs

A key mask restricts which keys other positions may read. A padding position acting as a query may still read valid keys and produce a nonzero output. Whether to retain, zero or discard that output depends on its later use. Masking pad keys does not automatically erase every padding row.

Mean pooling provides a separate failure mode. Including padding rows in its denominator can make a sentence representation depend on padded length. Even after zeroing those rows, dividing by total padded length dilutes the result. Define membership of the sum and the counting rule separately. For valid indicators mᵢ, a masked mean is Σmᵢhᵢ/Σmᵢ when the denominator is positive; the all-padding case needs its own policy.

The loss layer has another contract. Ignoring padding labels can remove their direct loss contribution, but cannot undo an earlier unmasked attention operation’s influence on valid representations. Repairing loss participation does not replace repairing visibility in feature computation.

Define the fully padded case explicitly

When a query has no valid keys, the ordinary softmax normalization set is empty. An arbitrary large negative constant is not a universal solution. If every score becomes the same finite number, normalization can still assign uniform weights.

An application may reject content-free samples or implement and verify an explicit zero-output path. The choice must agree with pooling, loss and batching contracts. Different backends, versions and mask representations can handle edge cases differently. This article does not promise that every implementation returns NaN or that every implementation returns zero.

Real tests should cover no padding, partial padding, fully padded input and unequal lengths within a batch. Inspect expected loss and gradients as well as output. A forward pass that does not raise an error is insufficient evidence of correct padding semantics.

Trace participation through the layers

Start with token IDs and valid lengths, then inspect embedding output, positional transformations, attention visibility, valid-query outputs, pooling and loss reduction. Recording which positions participate at each stage localizes padding-related differences to a specific operation.

The earlier SDPA inference dropout article examines a randomness setting. That setting also needs control in this regression test, whose main concern is normalization membership. Construct minimal examples for the two issues separately; changing both at once makes attribution harder.

If padding_idx is already set but identical content still changes with padded length, inspect the attention and reduction pipeline next. If this counterexample and layered validation method help isolate a bug, voluntary support through the section below is welcome.

Sources and backfill note

API behavior was checked against PyTorch 2.8 documentation for Embedding, SDPA and MultiheadAttention. Both figures, the uniform-attention example and twelve checks are original material. No PyTorch or GPU kernel ran. The article retains October 18, 2026 at 19:00 Beijing time as its planned date and was backfilled on October 19, 2026.