Crabhaus

What it is

A token is the atomic unit an LLM reads and writes: usually a chunk of characters (a short word, a word-piece, or even a single byte), not a whole word. Tokenization chops raw text into these units and maps each one to an integer ID drawn from a fixed vocabulary.

How it works

Most LLMs use byte-level Byte-Pair Encoding: training starts from raw bytes and repeatedly merges the most frequent adjacent pair into a new symbol, building a vocabulary of roughly 50k to 200k merges. At inference the same rules greedily form the longest known tokens, so common words become one token while rare words, code, or other languages split into many.

A concrete example

"tokenization" splits into "token" plus "ization" (two tokens) while "the" is one. A number like "12345" can break into digit-chunks, and an emoji or a non-English word may cost several tokens each. Rule of thumb: one English token averages about four characters, so roughly 750 words is near 1,000 tokens.

Why it matters

Tokens, not words, are the unit of billing, context windows, and rate limits: a 128k context is 128k tokens, and you pay per token in and out. Token boundaries also explain quirks like miscounting letters or spelling a word backwards, since the model never sees the characters inside a token.