Crabhaus

What it is

An LLM that accepts images alongside text. Instead of a separate OCR or captioning system bolted onto a chatbot, the image becomes part of the model's input sequence, so the same transformer that reasons over words reasons over what is in the picture.

How it works

A vision encoder slices the image into a grid of small patches - 16x16 pixel tiles - and turns each patch into a vector shaped like a text-token embedding. A projection layer maps those vectors into the LLM's embedding space, so the image arrives as a run of extra tokens ahead of your prompt. Attention spans both kinds of token; more resolution means more patches, and more tokens.

A concrete example

Send a screenshot of an error dialog to Claude or GPT and ask what to click. The encoder turns it into a few hundred patch tokens and the model reads button labels and the stack trace straight out of them - no separate OCR pass. Every screenshot-driven computer-use agent runs this loop on every step.

Why it matters

Image tokens are billed like text tokens, so resolution is a cost dial: a full-screen screenshot can cost more than the prompt around it, and an agent that screenshots every step pays it per step. Cropping and downscaling are often the cheapest optimization available - and VLMs have quietly displaced classic OCR pipelines along the way.