Crabhaus

What it is

A transformer design that swaps each dense feed-forward layer for a bank of parallel subnetworks ('experts') plus a small learned router. Each token activates only a few experts, so the parameters used per token are a small fraction of what the model contains - the gap behind 'active vs total' spec sheets.

How it works

At each MoE layer the router scores all experts for the current token and sends it through the top-k - often 1 to 8 out of up to a few hundred. Only those experts run; the rest sit idle for that token, and load-balancing keeps tokens spread evenly. The trade is memory for compute: every expert stays resident, so serving needs RAM for total parameters but FLOPs only for active ones.

A concrete example

DeepSeek-V3 holds 671B total parameters but routes each token through roughly 37B - about one-eighteenth of the model - by selecting 8 of 256 routed experts per layer. Its per-token serving compute tracks a mid-size dense model, not its headline size.

Why it matters

Most frontier-scale models are now sparse: Kimi K3 is a 2.8T-parameter MoE, and DeepSeek's V4.1-Flash carries a 552B backbone. Reading active-vs-total correctly tells you a model's real inference cost, why trillion-parameter open weights are economically possible, and why API prices don't scale with parameter counts.