The √d factor in attention controls the scale of scores entering softmax. It counters the growth in dot-product variation as dimensionality increases. It neither bounds every score to a fixed interval nor guarantees stable training.

Why the square root appears

A query and key each contain d components. Under the idealized assumptions of zero means, unit variances and the independence required by the derivation, their dot product sums d products. Its variance is d and its standard deviation is √d. Dividing by √d brings the variance to one.

Learned components can be correlated and need not have unit variance. These assumptions explain a scaling choice; they are not measured statistics for every real model.

What scaling does to the weights

Consider dot-product scores 8 and 0 with d=64. Without scaling, the first softmax weight is about 0.999665. Scaling gives scores 1 and 0, reducing that weight to about 0.731059. This illustrates a change in fixed scores, not a training run.

For two scores separated by Δ, the first weight is 1/(1+exp(-Δ)). Large gaps push weights toward zero or one. Diagonal entries of the softmax Jacobian are p(1-p), so extreme weights have smaller local derivatives. Other paths still affect the full network gradient.

Scaling and numerical stability differ

Subtracting the largest score before softmax preserves normalized weights and reduces the risk of exponential overflow. It preserves score differences, so it does not replace division by √d.

Temperature also scales scores, but its purpose and location matter. Attention key-dimension scaling differs from temperature sampling over vocabulary logits during generation.

import math

def softmax(xs):
    m = max(xs)
    es = [math.exp(x-m) for x in xs]
    return [e/sum(es) for e in es]

raw = softmax([8.0, 0.0])
scaled = softmax([8.0/math.sqrt(64), 0.0])
assert raw[0] > 0.999
assert math.isclose(scaled[0], 0.7310585786300049)
assert math.isclose(sum(scaled), 1.0)
assert all(math.isclose(a,b) for a,b in
           zip(scaled,softmax([101.0,100.0])))
print(round(raw[0],6), round(scaled[0],6))

Applying the example to a model

The Python example was executed, checking the weight sum, scaled values and translation invariance. It loads no model and measures no GPU performance; it is not a quality or throughput benchmark.

Attention Is All You Need first appeared in 2017. This is a historical method, not a new release. Check whether your implementation already applies scaling to avoid dividing twice. For the distinction between query and KV heads, see GQA.

中文版

Sources

Attention Is All You Need

Prepared on October 12, 2026; the article date retains its planned slot.