Crabhaus

What it is

Attention is a way for a model to decide which other tokens matter while it updates the representation of each token. A transformer is the architecture that repeats this attention-and-update process in stacked blocks, letting every position exchange information without processing the sequence one token at a time.

How it works

Each token becomes a query describing what it seeks, a key describing what it matches, and a value carrying information. The model compares one query with every key, turns the matches into weights, and blends the values. Multiple heads learn different relationships in parallel. Feed-forward layers update each token, while position information preserves order.

A concrete example

In “The animal did not cross the street because it was tired,” the representation for “it” can place more weight on “animal” than “street.” Another head may track the negation in “did not.” After several layers, each token carries context gathered from the rest of the sentence instead of standing for an isolated word.

Why it matters

Transformers made training highly parallel and let models combine evidence across long contexts. The same basic block now powers language models, vision systems, audio models and multimodal agents. Understanding attention also explains practical limits: every token-to-token comparison costs memory and compute, so long contexts need caching, sparse attention or other optimizations.