RoPE rotates a query and a key using their individual positions, yet their dot product can depend explicitly on relative position alone. The key is cancellation: multiplying two rotation matrices leaves the difference between their angles. That identity also provides concrete tests for position numbering, rotation direction, and cache offsets.

Start with a two-dimensional rotation, extend it by coordinate pairs, and test the resulting properties with a runnable example. 阅读中文版.

State the property before implementing it

Fix two projected content vectors q and k at positions m and n. Shifting both positions by the same amount preserves their relative distance. We want the explicit positional transformation to preserve the pair score under that shared shift. This does not require unrelated tokens with different content to have equal scores.

RoPE originates in the RoFormer paper. The original text checked here is arXiv v5, dated November 8, 2023; the first version appeared in 2021. This is a foundational implementation explanation, not a new research announcement. The numerical examples and tests below are constructed for the identity, rather than borrowed model results.

Why a rotation leaves an angle difference

Let R(a) rotate a two-dimensional vector: (x,y) becomes (x cos a−y sin a, x sin a+y cos a). Its transpose is the inverse rotation, giving R(a)ᵀR(b)=R(b−a). This is a property of planar rotations, independent of a trained model.

Using column vectors, rotate q by mθ and k by nθ. Their dot product becomes (R(mθ)q)ᵀ(R(nθ)k)=qᵀR((n−m)θ)k. Only the position difference remains. Swapping query and key or changing notation changes the sign convention; copying an m−n expression without its vector convention can conceal a mistake.

Extend the construction across coordinate pairs

Split an even-dimensional vector into two-dimensional subspaces, each with its own frequency. The same relative-position identity holds within each pair, and the full dot product is their sum. This example pairs adjacent dimensions: (0,1) and (2,3) for a four-dimensional vector.

The code uses θⱼ=10000^(−2j/d), with zero-based j and rotary dimension d. Earlier pairs change phase faster. Real models may rotate only part of a head or use other frequency and scaling settings. The Transformers rotary utilities documentation describes configuration variants; this teaching default is not a checkpoint-independent prescription.

There is no need to materialize a dense d-by-d matrix. Each pair needs a small fixed amount of arithmetic, so rotating one vector takes work linear in d. That statement concerns the rotation operation, not the complexity of a full attention layer or generation request.

Turn the identity into tests that can fail

Output shape alone will not catch an offset mistake. Check norm preservation, agreement between two-sided rotation and relative rotation, invariance under a shared position shift, and explicit rejection of unsupported odd dimensions.

# Small adjacent-pair RoPE model, not a model inference implementation.
import math

def rotate(v, pos, theta=10000.0):
    if not v or len(v) % 2:
        raise ValueError("nonempty even dimension required")
    out = []
    for j in range(len(v)//2):
        angle = pos * theta ** (-2*j/len(v))
        c, sn = math.cos(angle), math.sin(angle)
        x, y = v[2*j:2*j+2]
        out.extend([x*c-y*sn, x*sn+y*c])
    return out

def dot(a, b):
    return sum(x*y for x,y in zip(a,b))

q, k = [1.,2.,3.,4.], [2.,-1.,0.5,3.]
for m,n in [(0,0),(2,7),(7,2),(-3,8)]:
    lhs = dot(rotate(q,m), rotate(k,n))
    rhs = dot(q, rotate(k,n-m))
    assert math.isclose(lhs, rhs, abs_tol=1e-10)
    shifted = dot(rotate(q,m+23), rotate(k,n+23))
    assert math.isclose(lhs, shifted, abs_tol=1e-10)
assert math.isclose(dot(q,q), dot(rotate(q,9),rotate(q,9)))
try:
    rotate([1.,2.,3.],0)
except ValueError:
    pass
else:
    raise AssertionError("odd dimension was accepted")
# A single-frequency score is periodic, not monotonically decreasing.
assert math.cos(2*math.pi) > math.cos(math.pi)
print("relative offset, shared shift, norm, and odd dimension checks passed")

The code was executed locally for equal positions, positive and negative offsets, shared shifts, norm preservation, and odd input dimensions. It uses small vectors and double-precision arithmetic, not model inference. Its tolerance applies to these conditions and is not a universal acceptance threshold for half-precision kernels.

Why norm preservation alone is insufficient

If query and key use different frequency settings, each transform can remain a valid rotation and pass its norm test. Their combined phase is no longer a shared frequency times the position difference. A shared position shift can then change the dot product. Orthogonality alone does not establish a common coordinate contract.

An additional local four-dimensional counterexample uses base 10000 for the query and 1000 for the key. Both preserve norms, but shifting both positions by 23 changes the score. Combine norm, relative-dot-product, shared-shift, and known-value checks: passing one invariant is not a proof of a correct implementation.

Relative position does not mean monotonic decay

For two-dimensional q=k=(1,0), the score is cos((n−m)θ). It is −1 at an angle difference of π and returns to 1 at 2π. This simple counterexample rules out interpreting RoPE as a mandatory monotonic distance penalty for every individual pair of tokens.

Multiple frequencies and learned content vectors can produce richer patterns, but a particular score still depends on content and phase. Aggregate properties or statistical trends should not be rewritten as pointwise monotonic guarantees. The counterexample clarifies that boundary without denying the role of positional encoding.

Cache positions must match logical coordinates

During incremental generation, a token’s array index need not be its logical sequence position. After processing an existing prefix, a one-token input sits at index zero in its small tensor. That does not automatically make it position zero in the whole sequence. The inference protocol determines the position.

An inspectable check holds content and logical positions fixed, then compares keys rotated in one pass with keys rotated in segments. Restarting every segment at zero while retaining old keys at their original positions changes relative angles. Such an error can preserve every tensor shape.

If a cache already stores rotated keys, applying the transformation again changes their phase. Check the implementation’s storage convention rather than inferring it from a variable name. This is a comparison method, not a claim that every framework caches the same representation.

Coordinate layout must agree with the checkpoint

Adjacent-pair and split-half layouts can represent the same mathematical structure under a consistent coordinate permutation. Their numerical layouts differ. Weights, data order, and rotation must agree before outputs can be compared; replacing only a rotation function may break compatibility.

For debugging, fix a small vector and list each coordinate pair, frequency, and angle. Compare individual rotations first, dot products second, and full attention last. This separates layout mistakes from position-numbering mistakes instead of guessing from generated prose.

The opening derivation also fixes the content parts of q and k. Moving an input through a real multilayer model can change masks, context, and earlier representations. A local identity therefore does not establish identical complete-model output under arbitrary shifts or extensions.

A defined long position is not validated long-context quality

Trigonometric functions can be evaluated beyond training length, but a defined formula is only an implementation condition. Unseen phase combinations, numerical precision, and task requirements still need model-specific evidence. Increasing a maximum position or changing a scaling setting and obtaining text does not establish preserved quality.

Validate in two stages: use identities like these to rule out implementation errors, then evaluate the model at the intended lengths and tasks. A failed identity points back to position or layout; a passed identity does not remove the need for behavioral evaluation. If these checks save you time debugging RoPE, voluntary support below is welcome.

Sources and verification scope

RoFormer, arXiv v5, and Transformers rotary utilities. Diagrams, counterexamples, and tests are original teaching content, without training or GPU measurements. Backfilled on the evening of October 15; the article date retains the planned 14:00 Beijing slot.