Crabhaus

What matters in AI, and why.

Wed 7 Oct · Issue 085 · 3 min By Vatsal Mishra

Issue 085 · Wed 7 Oct

Europe's 1T chonk enters the open-weights race

01LabsMistral Large 4: 1T preview, weights end of month
02AgentsDecision models arrive: OpenAI Decisions API beta
03BizDeepSeek nears $12B+ round before early-2027 IPO

The one that matters

Mistral previews Large 4: a 1T 'chonk' aimed at the open-weights crown

ML4 is in public preview on Mistral Studio: 1T parameters, 49B active, natively multimodal, trained from scratch on 3,800 Grace Blackwell GPUs in Mistral's European datacenters. Mistral says it significantly outperforms any open-weight model built in the US or Europe and is competitive with the strongest open models globally; weights land by end of October.

Why it matters

A day after Reflection's Beam, the open-weights race gets a 1T European entry trained entirely outside US and Chinese clouds - sovereign AI as a download link rather than a policy deck. For builders the number to watch is 49B active parameters: that is the serving bill. Caveats: preview-only API today, benchmarks are Mistral's own, and 'end of month' weights have slipped at other labs before.

Learn one thing

Evaluating RAG: retrieval metrics and faithfulness

Hand-drawn two-layer RAG evaluation diagram. Left: documents flow through a retriever into three chunks, with a 'Layer 1: Retrieval' gauge and dials for context recall and precision. Right: the chunks plus a question enter an LLM whose answer splits into atomic claims checked against the chunks, one marked unsupported, under a 'Layer 2: Faithfulness' gauge. Bottom: an outcome matrix.View full-size diagram ↗

What it is

A two-layer approach to measuring RAG systems. Layer one scores retrieval: did the system surface the chunks that actually contain the answer (context recall and precision). Layer two scores generation: is every claim in the answer supported by those chunks (faithfulness). A single end-to-end quality score blends the layers together and hides which one failed.

How it works

Retrieval gets IR metrics: context recall = what fraction of facts needed for a reference answer appear in the chunks; context precision = do relevant chunks outrank irrelevant ones. Faithfulness (as in Ragas) splits the answer into atomic claims and checks each against the retrieved context. Wrong chunks + faithful answer: retrieval bug. Right chunks + unsupported claims: generation bug.

A concrete example

A support bot answers 'What's the free-tier rate limit?' from the enterprise pricing page: the reply is perfectly faithful to its chunk and still wrong - context recall catches it, so you fix chunking or embeddings. If the right page was retrieved but the reply cites a number that appears nowhere in it, faithfulness drops - you fix the prompt or grounding, not the index.

Why it matters

Teams tune prompts for weeks when the real failure is retrieval, because end-to-end scores can't say which layer broke. Two-layer evals turn 'the bot is sometimes wrong' into a routable bug report, and they make regressions visible when you swap an embedding model, reranker, or chunk size.

What people get wrong: That a faithful answer is a correct answer. Faithfulness only checks grounding in what was retrieved: an answer fully supported by stale or off-target chunks is still wrong, and a correct answer can be unfaithful when the model answered from pretraining while retrieval contributed nothing - which breaks once questions leave the training distribution.

See every explainer, in order →

Today's top 5

BizDeepSeek is close to raising $12B+ in a round that could reach ~$14.9B, with Tencent and CATL among top backers, before an early-2027 IPO.

If the round closes near $14.9B, DeepSeek enters its IPO year with more fresh capital than most Western labs' lifetime raises, backed by Tencent and a battery giant rather than US cloud credits. Open-weight releases funded by a war chest that size reset the economics every US open-model effort gets measured against.

AgentsMeta, Walmart, Shopify, Stripe, and Sierra published the Personal Agent Protocol to standardize how AI agents transact with businesses.

Agent-to-business commerce is currently bot-blocking, scraping, and one-off integrations; a protocol backed by Meta plus the merchant stack (Walmart, Shopify, Stripe) is the first serious attempt to standardize identity, authorization, and payments for agents. Which labs adopt it decides whether this becomes the HTTP of agent commerce or another consortium PDF.

AgentsOpenAI's Decisions API hit public beta: typed answers (probabilities, choices, rubric scores) about 10x faster than the Responses API.

Classification, routing, and prioritization inside agent loops don't need prose, they need a fast typed verdict. A dedicated endpoint returning probabilities ~10x faster than Responses - landing alongside AWS's open Decider 2B and TypeSafe's Jev - says decision models are hardening into a real product category. Worth a look before you burn full LLM calls on yes/no checks.

ResearchOpenAI published hundreds of math results from an internal frontier model, with Lean formalizations and per-result compute disclosures.

Lean formalizations make many of these results machine-checkable rather than trust-the-lab claims, and the compute disclosures (stated in ChatGPT Pro usage terms) give a rare cost yardstick for frontier reasoning. The open question is review at scale: hundreds of machine-generated manuscripts just landed on a field that peer-reviews in months, not minutes.

LabsGoogle DeepMind released EmbeddingGemma 2, a 740M Apache-2.0 model embedding text, code, images, video, and audio in one shared space.

One shared embedding space across five modalities at 740M params and Apache 2.0 makes multimodal retrieval cheap enough to run on your own hardware - the missing piece for local RAG over screenshots, docs, and recordings. It also pairs with today's Learn One Thing: swapping in a new embedding model is exactly when you re-run retrieval evals.

See the full wire feed →

Quick check

Was this issue useful?

Get Crabhaus in your inbox

A concise daily briefing on the AI developments worth understanding.