Crabhaus

What it is

A vector database is a store built around one query: given an embedding, find the stored embeddings closest to it. Its speed comes from an approximate nearest-neighbor (ANN) index, a data structure that returns near-matches in milliseconds by accepting a small chance of missing the true closest vector.

How it works

HNSW builds a layered graph: every vector links to a few near neighbors, and upper layers keep only sparse long-range links. A query enters at the sparse top, greedily hops toward whichever neighbor is closest, drops a layer, and repeats; at the dense bottom a small beam search gathers the final candidates. Each lookup touches a few hundred nodes, so search time grows roughly logarithmically.

A concrete example

A RAG app with 5,000 document chunks needs none of this: a NumPy dot product across the whole array answers in about a millisecond from a flat file in memory. The same app at 50 million chunks, with per-user filters and constant inserts, wants pgvector or a dedicated store, where an HNSW index answers in single-digit milliseconds while brute force would take seconds per query.

Why it matters

Retrieval sits inside every RAG and agent-memory loop, so its latency and cost multiply across every query served. Knowing where the crossover sits protects you from both failure modes: shipping brute-force search that collapses at scale, and bolting a vector database onto a 10k-chunk side project that a flat array would serve faster with zero ops burden.