Home · ELI5

Word order in attention

How Does RoPE Mark Position?

RoPE turns query and key arrows by their token positions so their match includes how far apart they are.

Position turns the arrows

Each step along the token sequence turns this pair another 30 degrees.

Q: query (solid)K: key (dashed)
A query and a key rotate with their token positions Frozen example: query at position zero, key at position two, relative angle 60 degrees, pair score 0.500. 0°90° Before rotation: both point right Q K token sequence →
Query position
0 · 0°
Key position
2 · 60°
Relative angle
60°
Pair score: cos(angle)
0.500

Paused at shift 0.

Illustrative model: fixed, identical unit arrows; 30° per token is a toy speed. This score is one component pair's dot product.

What these arrows stand for
  • Attention compares a query with a key. Each is a list of numbers made from token content.
  • Two numbers form one arrow. RoPE rotates these pairs inside queries and keys. The original Transformer adds position encodings to token embeddings.
  • This picture holds the unrotated arrows fixed to isolate position. Actual token content also changes the match.

The gap changes the match

For these fixed arrows, a wider gap changes the angle between them and their pair score.

1 token · 30°4 tokens · 120° score 0.866score −0.500
Why a score can be negative
  • For unit arrows, the dot product measures alignment: 1 for the same direction, 0 at a right angle, and −1 for opposite directions.
  • This pair score contributes to an attention score. Attention combines many pairs, scales the result, and converts scores into weights; the displayed number is not an attention probability.

Move both, keep the gap

Adding three to both positions turns both arrows equally, keeping their relative angle and pair score.

Q 0 · K 2Q 3 · K 5 gap 2 · angle 60°gap 2 · angle 60° same pair score: 0.500
When equal gaps give equal scores
  • The identity is R(m)ᵀR(n) = R(n − m): the two rotations reduce to their difference.
  • Equal gaps preserve the dot product when the unrotated query and key content stays fixed. Different content can give different scores at the same gap.

Several pairs turn at different speeds

RoPE uses fast and slow rotations across many pairs to represent position at several scales.

Faster: 30° / tokengap 4 → 120° Medium: 15° / tokengap 4 → 60° Slower: 5° / tokengap 4 → 20°Toy speeds, same four-token gap
Repeats and real context limits
  • One rotating pair repeats after a full turn. At the toy speed of 30° per token, positions 12 tokens apart line up again.
  • The original RoPE schedule uses θᵢ = 10000^(−2i/d) radians per token for pair i, where i starts at zero and d is the query/key dimension.
  • Multiple frequencies give different positional scales. Reliable use of long context also depends on training and the model; rotation alone does not guarantee it.

Sources: RoFormer, §3.2 (rotations and relative position); Attention Is All You Need, §3.5 (added position encodings). Related: What Is Attention?

Back to top