L2 normalization can hide an incorrect mean-pooling operation, but it cannot generally repair one. A positive rescaling of a nonzero sentence vector preserves its unit direction. Padding states that change the numerator can change that direction, and normalization keeps the error. Testing only the final cosine similarity can therefore miss a pooling bug.

We will hold the token states fixed and compare zero padding with nonzero padding. The resulting regression test checks the pooling contract without depending on a model’s semantic scores. 中文版.

Attention and pooling select different participants

An attention mask controls which positions may supply information during attention. A pooling mask selects which output rows contribute to the sentence vector and its count. Supplying a mask to the first operation does not justify averaging every row in the second. In particular, it does not establish that every padding output state is zero.

The ordinary mean branch in Sentence Transformers v5.1.0 Pooling expands the attention mask across the hidden dimension, uses it in the sum, and divides by a masked count with a lower bound. The implementation also supports other pooling modes. Our formula concerns ordinary mean pooling without additional token weights.

Let H have shape B × L × D and the mask have shape B × L. For one example, hᵢ is a D-dimensional token state, mᵢ is either zero or one, and n = Σmᵢ. When n > 0, the mean is u = Σ(mᵢhᵢ) / n. Broadcasting along the last dimension applies one participation decision to every feature of a token, rather than giving each feature an unrelated valid length.

The earlier article on padding_idx and attention masks explains how a zero embedding can still affect an attention denominator. Here we move downstream to pooling and normalization. The useful distinction is between a scaling error and a direction error.

A wrong denominator can leave the direction unchanged

Take two valid states, (2, 0) and (0, 2). Their correct mean is u = (1, 1). Append a zero padding row and average all three rows: v = (2/3, 2/3). The raw vectors differ, but v = (2/3)u.

Define N(x) = x / ‖x‖₂. For nonzero x and positive c, exact arithmetic gives N(cx) = N(x), since ‖cx‖₂ = c‖x‖₂. Both means therefore normalize to (1/√2, 1/√2). A test that compares only normalized outputs passes even though the averaging denominator is wrong. Normalization here happens after pooling. Normalizing each token first and then averaging is a different operation, outside this cancellation argument.

The same reasoning applies whenever the numerator remains the valid sum S but the code divides by padded length L instead of valid count n. The incorrect result S/L equals (n/L)u. Zero padding makes the example obvious, but the real condition is an unchanged numerator and a positive scaling factor.

This is not a reason to retain the bug. A pipeline may use the unnormalized vector for a dot product, a norm threshold, or another computation where scale matters. Zero vectors have no unit direction, and epsilon floors near zero also affect concrete normalization functions. The cancellation above is an exact mathematical statement under explicit conditions, not a promise about every floating-point implementation. See epsilon in cosine similarity for that boundary.

A contaminated sum survives normalization

Keep the two valid states and change only the padding row to (6, 0). Correct masking still yields (1, 1). An unmasked mean yields ((2 + 0 + 6)/3, (0 + 2 + 0)/3) = (8/3, 2/3), which points in the direction of (4, 1). It is no longer a positive multiple of the intended vector.

Against query q = (1, 0), the correct cosine score is 1/√2, approximately 0.7071. The contaminated score is 4/√17, approximately 0.9701. Keep the query and a second document unchanged. Suppose that document has unit vector (0.8, 0.6), giving score 0.8. The two documents now swap order. These are constructed teaching vectors, not measured retrieval results or an estimate of the error in any particular model.

A nonzero padding vector could happen to be collinear with the valid sum, so nonzero padding alone does not guarantee a direction change. The narrower conclusion matters: once excluded states enter the numerator, positive-scale cancellation no longer guarantees correctness. Inspect the sum and its direction rather than relying on normalization to remove unwanted information.

Twenty-four checks include the hidden failure

The following standalone Python program uses only the standard library. Fraction preserves the exact rational means; normalization uses floating-point arithmetic with a tolerance. It tests reductions over fixed hidden states. It does not load a Transformer, execute PyTorch, or test a GPU kernel.

from fractions import Fraction as F
from math import sqrt, isfinite, isclose


def mean_pool(rows, mask):
    if not rows or len(rows) != len(mask):
        raise ValueError('empty input or mask length mismatch')
    d = len(rows[0])
    if not d or any(len(row) != d for row in rows):
        raise ValueError('invalid hidden dimension')
    if any(m not in (0, 1) for m in mask):
        raise ValueError('mask must contain 0 or 1')
    if any(not isfinite(x) for row in rows for x in row):
        raise ValueError('hidden states must be finite')
    count = sum(mask)
    if not count:
        raise ValueError('no participating token')
    return tuple(sum((F(row[j]) for row, m in zip(rows, mask) if m), F(0)) / count
                 for j in range(d))


def unit(v):
    length = sqrt(sum(float(x) ** 2 for x in v))
    if length == 0:
        raise ValueError('zero vector has no direction')
    return tuple(float(x) / length for x in v)


def close(a, b):
    return all(isclose(x, y, rel_tol=1e-12, abs_tol=1e-12) for x, y in zip(a, b))


checks = 0

def check(condition):
    global checks
    assert condition
    checks += 1


def reject(fn):
    try:
        fn()
    except ValueError:
        check(True)
    else:
        raise AssertionError('expected ValueError')


valid = [(2, 0), (0, 2)]
correct = mean_pool(valid, [1, 1])
check(correct == (1, 1))
zero_pad = valid + [(0, 0)]
wrong_zero = mean_pool(zero_pad, [1, 1, 1])
check(wrong_zero == (F(2, 3), F(2, 3)))
check(wrong_zero != correct)
check(close(unit(wrong_zero), unit(correct)))
nonzero_pad = valid + [(6, 0)]
wrong_nonzero = mean_pool(nonzero_pad, [1, 1, 1])
check(wrong_nonzero == (F(8, 3), F(2, 3)))
check(not close(unit(wrong_nonzero), unit(correct)))
check(mean_pool(nonzero_pad, [1, 1, 0]) == correct)
# A correct numerator with the wrong length is still only a rescaling.
wrong_denominator = tuple(x * F(2, 3) for x in correct)
check(wrong_denominator != correct)
check(close(unit(wrong_denominator), unit(correct)))
# These are synthetic cosine scores against query (1, 0).
check(unit(correct)[0] < 0.8 < unit(wrong_nonzero)[0])
for n in (1, 3, 7):
    rows = valid + [(6, 0)] * n
    check(mean_pool(rows, [1, 1] + [0] * n) == correct)
    check(mean_pool(rows, [1] * len(rows)) != correct)
check(mean_pool(list(reversed(nonzero_pad)), [0, 1, 1]) == correct)
for rows, mask in [([], []), (valid, [1]), (valid, [0, 0]),
                   ([(2, 0), (1,)], [1, 1]), (valid, [1, 2]),
                   ([(float('nan'), 0)], [0])]:
    reject(lambda rows=rows, mask=mask: mean_pool(rows, mask))
reject(lambda: unit((0, 0)))
print(f'{checks} pooling checks passed')
print('correct:', tuple(map(float, correct)))
print('unmasked nonzero pad:', tuple(map(float, wrong_nonzero)))
print('synthetic query cosine:', unit(correct)[0], unit(wrong_nonzero)[0])

The local run printed “24 pooling checks passed”. The assertions distinguish averaging every row from preserving the valid sum but dividing by padded length. They compare raw means before checking directions and the synthetic ranking reversal. They append one, three, and seven excluded nonzero rows, confirming that correct masking preserves the result. Reversing rows and their corresponding mask also preserves participation.

The function rejects empty input, mismatched lengths, an all-zero mask, inconsistent hidden dimensions, nonbinary masks, and nonfinite hidden states. Unit normalization rejects a zero vector. These are deliberate contracts for this example. Because the implementation selects participating rows rather than multiplying a tensor by zero, it must not be used to infer tensor NaN behavior. Rejecting all nonfinite inputs avoids any claim that multiplying by zero removes NaN.

Connect the test to a real embedding pipeline

Start by capturing the hidden states and mask from one real forward pass. Test pooling in isolation: preserve the valid rows, append several synthetic nonzero padding rows, and mark them as excluded. A correct pooling result should stay consistent within a suitable floating-point tolerance. Nonzero padding is more diagnostic than zero padding because it reveals a missing numerator mask, whereas zero rows may produce only a scaling error that normalization hides.

Keep two assertions: the raw pooled value must match the reference mean, and the normalized output must match the reference direction. The second does not prove the first. Checking only that the output norm is close to one is weaker still: an incorrect direction can be normalized perfectly. Record the participating count, the sum, and the pre-normalization norm when investigating a discrepancy.

Then test the complete model with different padding lengths or batch companions. Hold tokenization, truncation, valid-token position semantics, and random factors constant. If valid hidden states already differ, investigate upstream computation before assigning all of the discrepancy to pooling. The isolated reduction test proves invariance only while valid input rows are fixed; it is not an unconditional invariance guarantee for the whole model.

Finally, inspect the model’s pooling contract. CLS, last-token, ordinary mean, and weighted mean are not freely interchangeable details. Do not discard tokens merely because they look like special symbols. Whether special tokens or a prompt prefix participate depends on the model and configuration. This small binary-mask example explains a reduction; it does not replace that wider contract.

All-padding input needs an application policy

When n = 0, the mean is undefined. The example raises an exception so that the missing-participant condition is visible. A library can bound the denominator to avoid a particular division by zero, but that does not turn an input with no valid content into a meaningful semantic representation. Rejecting, skipping, or explicitly marking such inputs are application choices that should be settled before similarity scoring and indexing.

An all-zero pooling mask is also different from an empty user string. A tokenizer may still produce special tokens, whose participation must be checked separately. Do not infer n from raw string length or treat the absence of an exception as proof that a sentence vector is useful.

Fix participation first, then check the normalization required by the similarity function. Separating scale from direction explains why a test can pass repeatedly before a different batch or padding state changes a retrieval order.

Sources and publication timing

The implementation reference is the ordinary mean branch of Sentence Transformers v5.1.0 Pooling.py. That version was released on August 6, 2025; it is not described here as the latest release. Both diagrams, the vectors, the ranking reversal, and the twenty-four checks are original teaching material. No real sentence-embedding model or retrieval-quality benchmark was run.

The article retains its planned date of October 23, 2026 at 14:00 Beijing time. It was actually published as a catch-up article on October 25, 2026.