Two nonzero vectors can point in exactly the same direction while cosine_similarity returns less than one. Check whether a norm is below eps. In the PyTorch 2.8 documented formula, each norm has its own lower bound, so an extremely small vector is not necessarily expanded to unit length. The score then reflects both direction and small-norm attenuation.

This affects the equivalence between normalization followed by a dot product and direct scoring, as well as thresholds and candidate order. Reproducible two-dimensional examples separate zero vectors, tiny nonzero vectors and ordinary unit vectors. A numerical safeguard returning zero is not treated as evidence about a geometric direction. 中文版.

Each norm is clamped separately

The PyTorch 2.8 cosine_similarity definition is s=(x·y)/(max(‖x‖₂,eps)max(‖y‖₂,eps)), with default eps=10⁻⁸. The two maximum operations apply to the two norms independently.

For nonzero x and y, rewrite this as s=cos(x,y)×r(x)×r(y), where r(v)=min(1,‖v‖₂/eps). Both factors equal one when both norms reach the lower bound, recovering geometric cosine. If either norm is smaller, the score magnitude is attenuated. This follows from algebra; it is not an empirical finding about an embedding model.

Choose the artificial value eps=0.01 to keep the arithmetic visible. With x=(0.005,0) and y=(1,0), the directions agree but the score is 0.5. Comparing x with itself produces 0.25. Scaling x to (0.01,0) restores a score of one against y. Direction stayed constant while the score changed, so positive rescaling is not invariant throughout this protected region.

A zero score has different possible causes

A zero vector has no defined geometric direction, and the ordinary cosine denominator is zero there. A positive eps can extend the numerical formula to return zero. That is an exceptional-input convention, not a statement that a zero vector makes a right angle with every other vector.

Genuinely orthogonal nonzero vectors also score zero. The dot product of (1,0) and (0,1) vanishes. Yet the squared distance from (1,0) to zero is one, while its squared distance to (0,1) is two. Equal scores can conceal different geometry and different input-quality states; a score alone cannot establish that an input is valid but unrelated.

Track validity separately in an application contract. Empty input, a failed upstream operation and an actual zero-valued model output are different states. Diagnosis can use authorized, non-sensitive status information without recording private source text. Whether an invalid vector is rejected, skipped or routed to an explicit fallback is a product decision that a denominator safeguard cannot make. A small norm is not automatically invalid. Keep numerical state and business validity separate rather than deleting valid examples merely to obtain more attractive scores.

When normalization followed by a dot product matches

The normalize reference defines v/max(‖v‖ₚ,eps), with defaults p=2 and eps=10⁻¹². In real-number arithmetic, applying p=2 normalization to the same vector dimensions with the same explicit eps, then taking a dot product, reproduces the separately clamped formula.

Using both functions’ defaults does not satisfy the same-eps condition. Take x=(5×10⁻⁹,0) and y=(1,0). A lower bound of 10⁻⁸ gives 0.5, whereas normalization with 10⁻¹² followed by a dot product gives one. Both follow their definitions. A floating-point tolerance alone does not explain a factor-of-two discrepancy in the intended real-number scores.

Another implementation clamps the product of norms as a whole. Even max(‖x‖‖y‖,eps²), using eps squared for compatible units, differs from clamping each norm independently. For x=(0.005,0), y=(1,0) and eps=0.01, the product-floor score is one while separate clamping gives 0.5. Preserving a parameter named eps is insufficient evidence that a scoring refactor preserves behavior.

Why the better-aligned candidate can rank lower

Let query q=(1,0), candidate A=(0.005,0) and candidate B=(0.8,0.6). Geometric cosine gives A=1 and B=0.8, so A is better aligned. With separate clamping and eps=0.01, A is attenuated to 0.5 while B remains at 0.8. The candidate order reverses.

No approximate-index miss is needed for this reversal; it follows from the scoring definitions themselves. If strict direction comparison is the design goal, inspect whether the small-norm policy matches it. If the service intentionally uses the protected score, construct the exact baseline with that rule rather than declaring the index wrong against a different formula.

The value 0.01 is chosen for readable ratios, not recommended as a production threshold. Nor does it imply that real models commonly emit such norms. Inspect authorized representative samples and the actual numerical type before deciding how important the boundary is for a deployment. A teaching counterexample establishes possibility, not an observed occurrence rate.

Thirteen checks isolate the default-parameter difference

The standard-library program implements the real-number formula using Python floats. It checks aligned score 0.5, self-score 0.25, the boundary, zeros, reversed ranking, default-eps differences and the product-floor counterexample. Its own contract rejects NaN and infinite coordinates. It calls neither PyTorch nor a retrieval index.

# Real-number formula reference with Python floats; no torch/index runtime.
from math import hypot,isfinite,isclose

def normalized(v,eps):
    if not v or not isfinite(eps) or eps<=0:
        raise ValueError('positive finite eps and nonempty vector required')
    if not all(isfinite(x) for x in v):
        raise ValueError('nonfinite coordinate')
    scale=max(hypot(*v),eps)
    return tuple(x/scale for x in v)

def score(a,b,eps):
    if len(a)!=len(b):
        raise ValueError('dimension mismatch')
    return sum(x*y for x,y in zip(normalized(a,eps),normalized(b,eps)))

eps=.01; q=(1.,0.); a=(eps/2,0.); b=(.8,.6)
assert isclose(score(a,q,eps),.5)
assert isclose(score(a,a,eps),.25)
assert isclose(score((eps,0.),q,eps),1.)
assert score((0.,0.),q,eps)==0.
assert score((0.,0.),(0.,0.),eps)==0.
assert score(a,q,eps)<score(b,q,eps)
assert score(a,q,1e-12)>score(b,q,1e-12)
assert isclose(score((5e-9,0.),q,1e-8),.5)
assert isclose(score((5e-9,0.),q,1e-12),1.)
# Clamping the product, even using eps squared, is a different formula.
product_floor=sum(x*y for x,y in zip(a,q))/max(hypot(*a)*hypot(*q),eps**2)
assert isclose(product_floor,1.)
assert not isclose(product_floor,score(a,q,eps))
for bad in [(float('nan'),0.),(float('inf'),0.)]:
    try:
        score(bad,q,eps)
    except ValueError:
        pass
    else:
        raise AssertionError('nonfinite coordinate accepted')
print('13 numerical and rejection checks passed')

These input checks belong to this example; they do not establish that a framework automatically rejects every nonfinite input. The program covers the moderate numbers and boundaries shown. It is not a production implementation for arbitrary precision, dimensions or extreme floating-point ranges. Representation of eps, operation order and rounding in lower precision require separate tests on the intended device.

Evidence to retain when changing scoring rules

First specify whether the system compares geometric cosine or a numerical score with a particular eps. Record the formula, eps, normalization axis, input dimension and computation type. Query processing, indexing and the offline baseline should follow the same contract. The statement already normalized does not capture enough information to reproduce a score.

For a real API, explicitly check the normalization axis and output shape as well, avoiding an accidental cross-batch comparison where row-wise scoring was intended. The tuple example here represents one equal-dimensional pair; it does not exercise broadcasting or tensor-dimension semantics. Include ordinary nonzero vectors, zeros, values below the bound, values on the bound and nonfinite inputs as separate test groups. For small-norm cases, inspect both scores and application validity. An invalid query can still pass through top-k and select records when every score is zero. Returning a list does not prove that the query was meaningful.

The earlier cosine and inner-product equivalence article covers unit vectors. This article adds the condition under which a protected normalization does not produce a unit vector. Inside that region, do not keep applying the unit-vector identity squared L2=2−2×dot product. Return to the distance expansion containing the actual norm of each operand. In the teaching example, protected A becomes (0.5,0) while the query remains (1,0). Their squared distance is 0.25, but incorrectly using the unit-vector identity gives one. The correct expansion is 0.25+1−2×0.5=0.25. Actual vector lengths explain the discrepancy; floating-point tolerance does not.

For an approximate-index comparison, establish this exact scoring contract before measuring search error. The top-k ties and recall discussion addresses tied results separately. Distinguishing numerical definitions, input validity and search behavior makes a discrepancy easier to localize than treating every ranking change as an index regression.

If these tiny-vector counterexamples help diagnose a normalization or threshold migration, voluntary support through the section below is welcome. The reusable result is a set of tests that checks the formula, exceptional-input state and ranking separately.

Sources and backfill note

cosine_similarity · normalize

Checked against versioned PyTorch 2.8 documentation available before the planned slot. Diagrams, vectors and calculations are original teaching material. Standard-library checks ran; no model, PyTorch or index ran. This backfill retains October 21, 2026, 19:00 Beijing as its article date, with actual publication time recorded separately.