Crabhaus

Archive

Every public edition, preserved as it was published.

Learn one thing

Every explainer, in order

A prerequisite-ordered walk through every explainer we've published.

1. Issue 005 · Sun 19 Jul

   Tokens and tokenization

   Explain how text becomes tokens (BPE), and why tokens — not words — drive cost, limits, and odd model behavior.

2. Issue 006 · Mon 20 Jul

   Context windows

   What a context window is, what happens on overflow, and why "bigger" isn't automatically "better".

   Builds on: Tokens and tokenization

3. Issue 007 · Tue 21 Jul

   Embeddings and vector similarity

   Meaning-as-vectors, cosine similarity, and what embeddings are (and aren't) good for.

   Builds on: Tokens and tokenization

4. Issue 008 · Wed 22 Jul

   Attention and the transformer, minus the math

   An intuition for how attention mixes information across tokens and why this architecture won.

   Builds on: Tokens and tokenization, Embeddings and vector similarity

5. Issue 009 · Thu 23 Jul

   Temperature, top-p, and why the same prompt gives different answers

   How next-token sampling works and how the knobs trade determinism vs creativity.

   Builds on: Attention and the transformer, minus the math

6. Issue 010 · Fri 24 Jul

   The KV cache

   Why long chats get slow and expensive, and how prefix reuse makes repeated context cheap.

   Builds on: Attention and the transformer, minus the math, Context windows

7. Issue 011 · Sat 25 Jul

   Base models vs instruction-tuned models

   What pretraining produces vs what post-training adds, and why "raw" models behave alien.

   Builds on: Attention and the transformer, minus the math

8. Issue 012 · Sun 26 Jul

   Quantization: running big models in small memory

   What 8-bit/4-bit quantization does, quality tradeoffs, and formats like GGUF.

   Builds on: Base models vs instruction-tuned models

9. Issue 013 · Mon 27 Jul

   System prompts and message roles

   How system/user/assistant roles are actually assembled into one token stream, and what system prompts can/can't enforce.

   Builds on: Context windows

10. Issue 014 · Tue 28 Jul

    In-context learning and few-shot prompting

    Why examples in the prompt change behavior without training, and when few-shot beats instructions.

    Builds on: System prompts and message roles

11. Issue 015 · Wed 29 Jul

    Chain-of-thought and reasoning traces

    Why "thinking out loud" improves accuracy, and what reasoning traces are (and whether to trust them).

    Builds on: In-context learning and few-shot prompting

12. Issue 016 · Thu 30 Jul

    Structured output and constrained decoding

    How JSON mode / schema enforcement actually constrain sampling, and failure modes.

    Builds on: Temperature, top-p, and why the same prompt gives different answers

13. Issue 017 · Fri 31 Jul

    Function calling: how tool use actually works

    The request/response loop behind "the model called a tool" — schemas in, arguments out, results back in-context.

    Builds on: Structured output and constrained decoding

14. Issue 018 · Sat 1 Aug

    Prompt caching economics

    How providers price cache reads/writes, and how to structure prompts to exploit it.

    Builds on: The KV cache

15. Issue 019 · Sun 2 Aug

    Streaming and perceived latency

    How token streaming works (SSE), and why time-to-first-token beats total time for UX.

    Builds on: Temperature, top-p, and why the same prompt gives different answers

16. Issue 020 · Mon 3 Aug

    Retrieval-augmented generation

    The retrieve-then-generate pattern, what it fixes (freshness, grounding) and what it doesn't.

    Builds on: Embeddings and vector similarity, Context windows

17. Issue 021 · Tue 4 Aug

    Chunking strategies

    Why document splitting quietly decides RAG quality; size/overlap/structure-aware tradeoffs.

    Builds on: Retrieval-augmented generation

18. Issue 022 · Wed 5 Aug

    Vector databases and approximate nearest-neighbor search

    What an ANN index (e.g. HNSW) does, and when you need a vector DB vs a flat file.

    Builds on: Embeddings and vector similarity, Retrieval-augmented generation

19. Issue 023 · Thu 6 Aug

    Hybrid search and rerankers

    Why keyword+vector beats either alone, and what a cross-encoder reranker adds.

    Builds on: Vector databases and approximate nearest-neighbor search

20. Issue 024 · Thu 7 Aug

    Context engineering

    Assembling the right context (instructions, retrieval, memory, tools) under a token budget — the discipline replacing "prompt engineering".

    Builds on: Retrieval-augmented generation, System prompts and message roles

21. Issue 025 · Fri 8 Aug

    Retrieval vs agent memory

    Contrast querying knowledge with maintaining evolving state across sessions.

    Builds on: Context engineering

22. Issue 026 · Sun 9 Aug

    Long context vs RAG: lost in the middle

    What degrades in million-token contexts and when stuffing beats retrieving.

    Builds on: Context windows, Retrieval-augmented generation

23. Issue 027 · Mon 10 Aug

    Grounding and citations

    How systems tie claims to sources, and why citation ≠ correctness.

    Builds on: Retrieval-augmented generation

24. Issue 028 · Tue 11 Aug

    The ReAct pattern

    Interleaving reasoning and acting, and why this simple pattern still underlies most agent frameworks.

    Builds on: Chain-of-thought and reasoning traces, Function calling: how tool use actually works

25. Issue 029 · Wed 12 Aug

    Designing tools agents can actually use

    What makes a tool schema learnable — naming, granularity, error messages as prompts.

    Builds on: Function calling: how tool use actually works

26. Issue 030 · Thu 13 Aug

    Model Context Protocol

    What MCP standardizes (servers, tools, resources), and what it deliberately doesn't.

    Builds on: Designing tools agents can actually use

27. Issue 032 · Sat 15 Aug

    Agent memory architectures

    Short-term vs long-term memory; files, vectors, and summaries as memory substrates.

    Builds on: Retrieval vs agent memory

28. Issue 033 · Sun 16 Aug

    Context compaction and summarization

    How agents compress transcript history — summarize-and-replace, tool-result clearing — to keep working past the context limit, and what quietly gets lost.

    Builds on: Context engineering, Agent memory architectures

29. Issue 034 · Mon 17 Aug

    The agent harness: the scaffolding around the model

    What actually surrounds a model in production — system prompt, tool surface, loop logic, compaction policy, permissions — and why the same model scores wildly differently in different harnesses.

    Builds on: The ReAct pattern, Designing tools agents can actually use, Context compaction and summarization

30. Issue 035 · Tue 18 Aug

    Planning and task decomposition

    How agents split goals into steps; plan-then-execute vs interleaved planning.

    Builds on: The ReAct pattern

31. Issue 036 · Wed 19 Aug

    Workflows vs agents

    When a deterministic pipeline beats an autonomous loop, and the cost of unnecessary autonomy.

    Builds on: Planning and task decomposition

32. Issue 037 · Thu 20 Aug

    Multi-agent orchestration patterns

    Orchestrator-worker, handoffs, debate; what coordination actually buys vs costs.

    Builds on: Workflows vs agents

33. Issue 038 · Fri 21 Aug

    Sub-agents and context isolation

    Why spawning a fresh-context worker beats one long transcript; token hygiene.

    Builds on: Multi-agent orchestration patterns, Context engineering

34. Issue 039 · Sat 22 Aug

    Human-in-the-loop approval gates

    Where to put approvals in agent flows; reversible vs irreversible actions.

    Builds on: Workflows vs agents

35. Issue 040 · Sun 23 Aug

    How coding agents work

    The edit-run-test loop, tests-as-reward, and why coding is agents' best-fit domain.

    Builds on: The ReAct pattern, Designing tools agents can actually use

36. Issue 041 · Mon 24 Aug

    Computer-use agents

    Screenshot-act loops, accessibility trees, and why GUI automation is much harder than API tool use.

    Builds on: The ReAct pattern

37. Issue 042 · Tue 25 Aug

    Evals 101: golden sets and regression tests

    Building a small labeled set and treating prompts/models like code under test.

    Builds on: Structured output and constrained decoding

38. Issue 043 · Wed 26 Aug

    LLM-as-judge and its pitfalls

    Using models to grade outputs; position bias, self-preference, calibration against humans.

    Builds on: Evals 101: golden sets and regression tests

39. Issue 044 · Thu 27 Aug

    Evaluating agents: trajectories, not answers

    Judging multi-step runs (tool choices, recovery, cost), not just final text.

    Builds on: Evals 101: golden sets and regression tests, The ReAct pattern

40. Issue 045 · Fri 28 Aug

    Benchmarks and contamination

    Why leaderboard scores mislead — training-data leakage, overfitting, saturation.

    Builds on: Evals 101: golden sets and regression tests

41. Issue 046 · Sat 29 Aug

    Reward hacking: when models game the objective

    Why optimizing a measurable proxy (RL rewards, evals, LLM judges) produces unintended behavior — specification gaming, eval cheating, and agents crossing scope to score well.

    Builds on: Evals 101: golden sets and regression tests, LLM-as-judge and its pitfalls

42. Issue 047 · Sun 30 Aug

    Hallucination: why models make things up

    The statistical roots of confident fabrication and what actually reduces it.

    Builds on: Temperature, top-p, and why the same prompt gives different answers, Grounding and citations

43. Issue 048 · Mon 31 Aug

    Guardrails and output validation

    Layered defenses — schema checks, classifiers, policy filters — and their failure modes.

    Builds on: Structured output and constrained decoding, Evals 101: golden sets and regression tests

44. Issue 049 · Tue 1 Sep

    Tracing and observability for LLM apps

    Spans/traces for prompts and tools; what to log to debug nondeterminism.

    Builds on: Function calling: how tool use actually works

45. Issue 050 · Wed 2 Sep

    Retries, fallbacks, and graceful degradation

    Handling provider errors, timeouts, and bad outputs without cascading failure.

    Builds on: Tracing and observability for LLM apps

46. Issue 051 · Thu 3 Sep

    Inference economics: what a token actually costs

    From GPU-hours to per-token pricing; why output tokens cost more; margin structure.

    Builds on: The KV cache

47. Issue 052 · Fri 4 Sep

    Batching, throughput, and latency

    Continuous batching and the throughput/latency tension in serving.

    Builds on: Inference economics: what a token actually costs

48. Issue 053 · Sat 5 Sep

    Speculative decoding

    Small-model drafting + big-model verification for faster inference at equal quality.

    Builds on: Temperature, top-p, and why the same prompt gives different answers, Batching, throughput, and latency

49. Issue 054 · Sun 6 Sep

    Running models locally: llama.cpp, GGUF, Ollama

    The local stack, its real hardware limits, and when local wins.

    Builds on: Quantization: running big models in small memory

50. Issue 055 · Mon 7 Sep

    Reasoning models and test-time compute

    Spending inference tokens to think; the new scaling axis and its cost implications.

    Builds on: Chain-of-thought and reasoning traces, Inference economics: what a token actually costs

51. Issue 056 · Tue 8 Sep

    Fine-tuning vs prompting vs RAG

    A decision framework for when each is the right tool.

    Builds on: Retrieval-augmented generation, Base models vs instruction-tuned models

52. Issue 057 · Wed 9 Sep

    LoRA and parameter-efficient fine-tuning

    How low-rank adapters make fine-tuning cheap and swappable.

    Builds on: Fine-tuning vs prompting vs RAG

53. Issue 058 · Thu 10 Sep

    Model distillation

    Training small models on big-model outputs; why it works and where it's contested.

    Builds on: Base models vs instruction-tuned models

54. Issue 059 · Fri 11 Sep

    Mixture-of-experts models

    Sparse activation — why "1T parameters" can be cheap to run; active vs total params.

    Builds on: Attention and the transformer, minus the math, Inference economics: what a token actually costs

55. Issue 060 · Sat 12 Sep

    Prompt injection

    Why untrusted content in context can hijack a model, and why there's no clean fix.

    Builds on: System prompts and message roles, Function calling: how tool use actually works

56. Issue 061 · Sun 13 Sep

    The lethal trifecta

    Private-data access + untrusted content + exfiltration channels — the combination that makes agents dangerous, and how removing one leg mitigates.

    Builds on: Prompt injection

57. Issue 062 · Mon 14 Sep

    Sandboxing agent execution

    Containers, VMs, and network egress controls for agent-run code — isolation as the backstop when instructions fail.

    Builds on: The lethal trifecta, Designing tools agents can actually use

58. Issue 063 · Tue 15 Sep

    Secrets management for AI agents

    Keeping credentials out of prompts, logs, and generated code.

    Builds on: Sandboxing agent execution

59. Issue 064 · Wed 16 Sep

    Red-teaming AI systems

    Structured adversarial testing — jailbreaks vs injections, manual vs automated attacks.

    Builds on: Prompt injection, Guardrails and output validation

60. Issue 065 · Thu 17 Sep

    Preference tuning: from RLHF to DPO

    How human preferences become a training signal; why DPO simplified the stack.

    Builds on: Base models vs instruction-tuned models

61. Issue 066 · Fri 18 Sep

    Constitutional AI and RLAIF

    Models critiquing models against written principles; AI feedback replacing human labels.

    Builds on: Preference tuning: from RLHF to DPO

62. Issue 067 · Sat 19 Sep

    Cascades: cheap-first model pipelines

    Trying a small model first and escalating on low confidence; cost/quality math.

    Builds on: Retries, fallbacks, and graceful degradation, Inference economics: what a token actually costs

63. Issue 068 · Sun 20 Sep

    LLM routing: sending each request to the right model

    How routers classify a request's difficulty and dispatch it to a cheap or frontier model, what signals they use (heuristics, classifiers, cascades), and where routing saves cost without hurting quality.

    Builds on: Inference economics: what a token actually costs, Cascades: cheap-first model pipelines

64. Issue 069 · Mon 21 Sep

    On-device and edge AI

    What actually runs on phones/laptops (NPUs, small LMs), and the privacy/latency/cost case.

    Builds on: Quantization: running big models in small memory, Running models locally: llama.cpp, GGUF, Ollama

65. Issue 070 · Tue 22 Sep

    Vision-language models (VLMs)

    How a model 'sees' an image (patches as tokens, projection into the LLM), and what that means for cost, OCR, and screenshot-driven agents.

    Builds on: Tokens and tokenization, Embeddings and vector similarity

66. Issue 071 · Wed 23 Sep

    GPUs, VRAM, and whether a model will fit

    How parameter count, precision, and context length translate into memory needs, and how to estimate if a model runs on given hardware.

    Builds on: Quantization: running big models in small memory, Running models locally: llama.cpp, GGUF, Ollama

67. Issue 072 · Thu 24 Sep

    Model file formats (safetensors, GGUF, checkpoints)

    What actually lives inside a model file, why safetensors replaced pickle-based checkpoints, and how GGUF ties quantization to loaders like llama.cpp.

    Builds on: Quantization: running big models in small memory

68. Issue 073 · Fri 25 Sep

    Prefill vs decode: the two phases of LLM inference

    Why prompt processing and token generation stress hardware differently, and how that shapes time-to-first-token, throughput, and cost.

    Builds on: The KV cache

69. Issue 074 · Sat 26 Sep

    Checkpointing and durable execution for agents

    How long-running agents survive crashes and restarts — persisted state, resumable steps, idempotent effects — and why 'just retry the whole run' breaks once agents take real-world actions.

    Builds on: Workflows vs agents, Retries, fallbacks, and graceful degradation

70. Issue 075 · Sun 27 Sep

    Idempotency: making agent actions safe to retry

    How idempotency keys, effect records, and replay-safe tool design prevent duplicate charges, messages, and writes when an agent retries.

    Builds on: Checkpointing and durable execution for agents, Retries, fallbacks, and graceful degradation

71. Issue 076 · Mon 28 Sep

    Code execution as agent action

    Why letting an agent write and run code can beat chaining dozens of tool calls — orchestration in one script, results filtered before hitting context — and the security bill that pattern runs up.

    Builds on: Function calling: how tool use actually works, How coding agents work, Sandboxing agent execution

72. Issue 077 · Tue 29 Sep

    Tool outputs are untrusted input

    Why search results, files, and API responses can carry prompt injection or malformed data, and how schema checks, provenance, sanitization, and least privilege contain them.

    Builds on: Prompt injection, Designing tools agents can actually use, Guardrails and output validation

73. Issue 078 · Wed 30 Sep

    Attention variants: MQA, GQA, and MLA

    Why labs replaced full multi-head attention with grouped or latent variants, what each trades off, and how they shrink the KV cache that dominates serving memory.

    Builds on: Attention and the transformer, minus the math, The KV cache

74. Issue 079 · Thu 1 Oct

    Semantic caching

    Caching LLM responses by embedding similarity (not exact-match) to cut cost and latency, and where it silently returns wrong answers.

    Builds on: Embeddings and vector similarity, Prompt caching economics

75. Issue 080 · Fri 2 Oct

    Prompt versioning and management

    Why prompts should be versioned artifacts (not buried strings), and how prompt changes tie into evals and rollbacks.

    Builds on: System prompts and message roles, Evals 101: golden sets and regression tests

76. Issue 081 · Sat 3 Oct

    Open weights vs open source: what model licenses actually allow

    The difference between open weights, open source, and restrictive model licenses, and what each lets a builder legally do.

    Builds on: Base models vs instruction-tuned models

77. Issue 082 · Sun 4 Oct

    Synthetic training data

    Why labs train on model-generated data, where it works (distillation, instruction data) vs where it degrades models, and what "model collapse" actually claims.

    Builds on: Base models vs instruction-tuned models, Model distillation

78. Issue 083 · Mon 5 Oct

    Positional encodings and RoPE

    Why transformers need position information, how rotary embeddings encode it, and why RoPE choices govern context-length extension.

    Builds on: Attention and the transformer, minus the math, Context windows

79. Issue 084 · Tue 6 Oct

    Why LLM outputs aren't reproducible (determinism and sampling randomness)

    Where nondeterminism actually comes from — sampling, floating-point/batching effects, and provider-side changes — and why temperature 0 still isn't a guarantee.

    Builds on: Temperature, top-p, and why the same prompt gives different answers

80. Issue 085 · Wed 7 Oct

    Evaluating RAG: retrieval metrics and faithfulness

    How to measure a RAG system in two layers — did retrieval surface the right chunks (recall/precision), and did the answer stay faithful to them — and why end-to-end vibes hide which layer failed.

    Builds on: Retrieval-augmented generation, Evals 101: golden sets and regression tests

81. Issue 086 · Thu 8 Oct

    Batch APIs: trading latency for half-price tokens

    How async batch endpoints work (submit, poll, 24h windows), why providers discount them ~50%, and which workloads (evals, backfills, classification) should never run on the realtime API.

    Builds on: Inference economics: what a token actually costs, Batching, throughput, and latency

82. Issue 087 · Fri 9 Oct

    Rate limits, queues, and backpressure

    How request and token limits propagate through an AI system, and how bounded queues, admission control, jitter, and load shedding stop retries from becoming an outage.

    Builds on: Retries, fallbacks, and graceful degradation, Batching, throughput, and latency

83. Issue 088 · Sat 10 Oct

    Token budgeting for long-running agents

    How to allocate a finite context budget across instructions, history, retrieval, tool results, and output while preserving the highest-value evidence.

    Builds on: Context engineering, Context compaction and summarization, Inference economics: what a token actually costs

84. Issue 089 · Sun 11 Oct

    Budgets and stop conditions for agents

    How step, token, time, and money budgets plus loop detection and explicit completion checks prevent agents from running forever or escalating cost without progress.

    Builds on: The agent harness: the scaffolding around the model, Evaluating agents: trajectories, not answers, Token budgeting for long-running agents