Crabhaus

What it is

A caching layer for LLM apps that reuses a stored response when a new request means roughly the same thing as an earlier one, not only when the text matches exactly. Requests are compared by embedding similarity, so paraphrases of a question can hit the cache even though their strings differ.

How it works

Each prompt is embedded into a vector. The cache keeps past prompt embeddings in a vector index beside their saved responses. A new prompt runs a nearest-neighbor lookup; score above the similarity threshold and the saved response returns with no model call. On a miss, the app calls the LLM and writes the new embedding and response back. Threshold, TTL, and eviction set how aggressive reuse is.

A concrete example

A support bot gets 'How do I reset my password?' and later 'what's the way to change a forgotten password'. An exact-match cache sees two different strings and pays for two completions. A semantic cache embeds both, finds them nearly identical in meaning, and serves the stored answer in milliseconds at near-zero marginal cost.

Why it matters

Production LLM traffic is heavily repetitive, so a tuned semantic cache turns a real slice of requests into fast, almost free hits - cutting token spend and median latency. It sits beside model routing and effort settings as a serving-layer cost lever that leaves quality on uncached requests untouched.