RoPE encodes position through rotations of queries and keys. Although the angles use absolute positions m and n, their inner product retains the angle difference n−m. Relative position can therefore enter attention scores without simply appending a position number.

Start with one two-dimensional rotation

Let R(a) rotate a vector counterclockwise by a radians. A vector (x,y) becomes (cos(a)x−sin(a)y, sin(a)x+cos(a)y). Rotation preserves length while changing direction.

Rotate the query by mθ and key by nθ. Their score is (R(mθ)q)ᵀ(R(nθ)k) = qᵀR((n−m)θ)k. The transpose of R(mθ) is R(−mθ), and multiplying rotations adds their angles. This is how relative position enters the inner product.

Adding seven to both positions rotates each vector by another 210°, but their angle difference stays 90°. The example score is unchanged. This holds with content vectors q and k fixed; changing real context can also change the model’s internal vectors.

Turn the identity into a runnable check

The Python example checks the relative-rotation identity, invariance to a common position shift, and preservation of length. It was executed and prints approximately zero. An absolute tolerance prevents rounding from being mistaken for an identity failure.

import math

def rotate(x, angle):
    c, s = math.cos(angle), math.sin(angle)
    return (c*x[0]-s*x[1], s*x[0]+c*x[1])

def dot(a,b):
    return sum(x*y for x,y in zip(a,b))

q, k = (1.0,0.0), (1.0,0.0)
theta = math.pi/6
m, n = 2, 5
score = dot(rotate(q,m*theta),rotate(k,n*theta))
relative = dot(q,rotate(k,(n-m)*theta))
shifted = dot(rotate(q,(m+7)*theta),
              rotate(k,(n+7)*theta))
assert math.isclose(score,relative,abs_tol=1e-12)
assert math.isclose(score,shifted,abs_tol=1e-12)
assert math.isclose(dot(rotate(q,m*theta),rotate(q,m*theta)),
                    dot(q,q),abs_tol=1e-12)
print(round(score,6))

This checks a single two-dimensional subspace. It runs no full Transformer and measures no long-document quality. Choosing θ=π/6 makes the arithmetic easy; it does not specify a particular model’s frequency configuration.

Higher dimensions use multiple angles

The original RoFormer work was submitted to arXiv in 2021. RoPE applies rotations in two-dimensional subspaces with different frequencies. With consistent pairing, frequencies and position conventions, the relative-rotation relation holds for each pair.

Channel layouts may differ: adjacent components or corresponding components from two halves can be paired. The convention must match the weights and library. First inspect a small vector, then verify position indices, frequency tables and position advancement with cached keys and values. A shared function name is not evidence of interchangeability.

Computable farther does not mean useful farther

A rotation formula can accept larger position indices. That is mathematical computability, not evidence that a model retrieves and reasons correctly beyond its training length. The three assertions above cannot establish long-context quality.

Evaluate the same retrieval task at multiple lengths and evidence locations, recording accuracy and latency. Also record the model version, position configuration, truncation rules and available KV-cache budget. Accepting a long input and reliably using all of it require separate checks.

Understanding why two rotations leave an angle difference is a useful starting point. Verify that local identity before making claims about a whole model’s context capability.

中文版

Sources

Original RoFormer preprint (2021); authors’ implementation. This is a principle explanation, not a benchmark reproduction or a new announcement.