Crabhaus

What it is

Chunking is the preprocessing choice that splits documents into retrievable units before a RAG system embeds and indexes them. Units may be fixed token windows, paragraphs, sections, semantic segments, or small children linked to larger parents.

How it works

Small chunks make individual facts easier to match but can sever headings, warnings, tables, and qualifiers. Large chunks preserve context but dilute embeddings and use more context-window space. Overlap repeats text across boundaries. Structure-aware splitting follows headings and paragraphs; parent-child retrieval embeds a precise child but returns its broader parent when selected.

A concrete example

A device manual has a Resetting section with a warning and five numbered steps. A blind token split can separate the warning from the procedure. A structure-aware splitter keeps the section intact, or embeds smaller step groups as children while returning the full parent section so the warning remains attached.

Why it matters

Retrieval can only return the units placed in the index. If an answer and its qualifier land in different chunks, better prompting cannot reliably restore the missing context. Tests also show that results vary by corpus, so teams should evaluate chunking on real questions before swapping models.