Golden sets
A small, versioned set of real cases that defines “working”.
What you’ll learn
Four ideas, one module
Three ways to grade
Deterministic checks, human review and LLM judges, and when to use each.
Failures and regressions
Name the failure type, then rerun the set on every change.
Quality, cost, latency
Choose what belongs in production, not what won the demo.
Applied exercise
Better, slower, pricier
Compare two architectures where one performs better but is substantially slower and more expensive. Decide what belongs in production.
- Architecture A
- Architecture B
- The decision
Cheat sheet
If you can’t name the number that would make you roll back, you don’t know it works
One page you keep: the questions to ask on every project, the rules of thumb, and a template you can use on your own product. All ten come with The Book.
Leave at the door
Five myths this module breaks
| Myth | What’s actually true |
|---|---|
| “LLM-as-a-judge is objective.” | It has biases. Calibrate it against human labels. |
| “I tried it a few times. It works.” | A few tries are anecdotes. Use a golden set. |
| “One accuracy number tells the story.” | Break failures down by type and by cost. |
| “The best model belongs in production.” | The best fit for quality, cost and latency does. |
| “Evals are a launch step.” | Rerun them on every prompt, model or data change. |
Before you move on
Can you call this one?
After a model upgrade, users say the assistant “got worse”.
What would you need in place to confirm or refute it in an hour? Name three pieces.
How we would answer
- A golden set real cases with expected behavior, agreed before the upgrade.
- A baseline stored scores for the old version on the same set.
- Graders deterministic checks, plus a judge calibrated against human labels.
- A failure taxonomy to label what “worse” means and which box it lives in.
- Logs of real requests to add the complaining users’ cases to the set.
- Cost and latency tracked too: “worse” is sometimes “slower”.
Every module ends on a question like this. Join the waitlist →
The Benchmark Saboteur
Remy Calder
stress-tests the tests themselves, exposing impressive scores that hide embarrassing failures.
Episode 09 · Building AI you can trust
Included with the course.
Coming soon · an AI character from the show, not a real researcher.
Course complete