Crabhaus

What it is

The signal transformers inject so attention knows token order — attention alone treats its input as an unordered set. RoPE (Rotary Position Embedding, from 2021's RoFormer paper) is the scheme nearly every current open-weights LLM uses, including the Llama and Qwen families.

How it works

Instead of adding a position vector to embeddings, RoPE rotates query and key vectors inside attention. Each vector splits into 2D pairs, each pair rotated by position times a per-pair frequency: fast-spinning pairs capture local order, slow ones long-range structure. Dot products then depend only on the distance between tokens, not absolute slots.

A concrete example

Context extension is mostly RoPE surgery. A model trained at 8K tokens has never seen the rotation angles 128K produces, so quality collapses out there. Position interpolation and YaRN rescale the frequencies so long sequences map back into familiar angles, then fine-tune briefly; YaRN reports ~10x fewer training tokens than earlier recipes.

Why it matters

A model card's context length is a property of how its RoPE frequencies were set and trained, not a free dial. A '1M context' variant is RoPE scaling plus long-sequence fine-tuning, and the quality of that work decides whether distant tokens are genuinely used or merely tolerated.