Crabhaus

What it is

Tracing records every step of an LLM app - model calls, tool calls, retrievals - as timed, nested spans linked into one trace per request. Each span carries the model name, token counts, latency, and errors, so a nondeterministic run can be reconstructed after the fact instead of guessed at from scattered logs.

How it works

Each unit of work opens a span; parent IDs nest spans into a trace, and W3C Trace Context headers carry the trace across services. OpenTelemetry's GenAI conventions standardize attribute names for model calls, token usage, and tool invocations, so any backend - Langfuse, Arize Phoenix, or a general APM - can render the same data as timelines, cost breakdowns, and eval scores.

A concrete example

A RAG agent answers slowly and wrong. The trace shows retrieval returned 8 chunks in 40ms, the first model call burned 2.1s on 6,000 input tokens (a bloated system prompt), and a tool call errored and silently retried three times. The logged prompt reproduces the bad answer; the fix is verified on the same trace view.

Why it matters

LLM apps fail nondeterministically and regress silently when a prompt, model version, or tool changes - and tokens and latency are the product's unit economics. Traces turn 'it sometimes gives weird answers' into a specific span with a diff, and the captured data doubles as the dataset for evals and regression tests.