Crabhaus

What it is

The traffic-control layer of an AI system. Providers cap requests per minute and tokens per minute with token-bucket counters, answering HTTP 429 when a budget empties. Backpressure is the discipline of passing that slow-down signal upstream through your queues and workers, instead of letting failed requests pile up and retry themselves into an outage.

How it works

Two budgets are metered at once: request count and token throughput, often charged against a max_tokens estimate before the call runs. On 429, a well-behaved client caps its own concurrency, holds pending work in a bounded queue, and retries with exponential backoff plus random jitter so workers never re-fire in sync. When the queue fills, admission control sheds new work at the front door.

A concrete example

A pipeline embeds 10,000 chunks with 100 parallel workers, trips the tokens-per-minute cap, and every naive retry loop hammers the API - throughput drops toward zero. The fix: a semaphore capping concurrency at 8, a bounded queue, jittered backoff that honors retry-after, and a dead-letter list for chunks failing three times. Same account, same limits - the job drains cleanly.

Why it matters

Agent systems are retry amplifiers: one user action fans out into dozens of model and tool calls. Without deliberate backpressure, a provider hiccup or a quota cut becomes a self-inflicted outage, and 429-driven retry loops quietly inflate the token bill. Concurrency caps, bounded queues, and jittered backoff are the difference between degrading gracefully and melting down.