RoPE: two pairs, two planes
Separate two 2D planes to compare rotation frequencies. Q and K rotate pair by pair; the vector stays 4D and V stays unchanged.
3D only arranges the planes. It adds no vector coordinates.
V does not enter RoPE [1,0,1,0] → [1,0,1,0]
Before division by √d, causal masking, or Softmax.
Use “Shift both +3” to capture a comparison. Then use “Shift K only +1” to change the position difference.
This captures the last button action. Moving a slider does not rewrite these values.
The full model has 32 coordinates per head: 16 independent 2D pairs. The 3D demo keeps its two pairs at d=4; this table lists the full model’s frequencies.
| Pair | Coordinates | ωᵢ |
|---|
Formula, scope, and sources
3D only arranges the planes. It adds no vector coordinates. Before division by √d, causal masking, or Softmax.
Q = K = [1,0,1,0] · V does not enter RoPE [1,0,1,0]
ωᵢ = theta^(−2i/d), theta=10000. Each pair (a,b) → (a cosα−b sinα, a sinα+b cosα).
The common-shift property assumes fixed Q/K content. Different position differences can produce the same dot product. Arrows show final direction; numeric angles retain the full rotation in radians.
The saved 2D example is verified before build. Its [1,0] content is repeated for the article’s 4D preset. Interactive values follow the course RoPE formula; V is a fixed unrotated comparison.
l13.json · tiny_lm/rope.py · ModelConfig · Static lesson figure
