← All modules

Module 08 · Evals

Building AI you can trust

A demo is not a system.

What you’ll learn

Four ideas, one module.

01

Golden sets

A small, versioned set of real cases that defines “working”.

02

Three ways to grade

Deterministic checks, human review and LLM judges, and when to use each.

03

Failures and regressions

Name the failure type, then rerun the set on every change.

04

Quality, cost, latency

Choose what belongs in production, not what won the demo.

Applied exercise

Better, slower, pricier

Compare two architectures where one performs better but is substantially slower and more expensive. Decide what belongs in production.

Open in your workbook
  • Architecture A
  • Architecture B
  • The decision

Cheat sheet · for your projects

If you can’t name the number that would make you roll back, you don’t know it works.

Ask on every project

  1. Which written cases define “working” for this feature, and who agreed on them?
  2. Which quality checks can plain code do before we reach for a model judge?
  3. Which failure type is most common, and what does each type cost us?
  4. What reruns the set when a prompt, model, index or tool changes?
  5. What p95 latency and cost per request can this use case afford?

Rules of thumb

  • Grade with code first, a calibrated judge next, and people to set the rubric.
  • Turn every production failure into a golden-set case before you fix it.
  • Compare pass rates over repeated runs, not single outputs, before judging a change.
  • Block a release when golden-set scores drop below the stored baseline.
Template

One-page evaluation plan

  • Scope: the feature, its users, and what a costly error looks like
  • Golden set: sources (logs, tickets, past failures), size, hard cases, who maintains it
  • Graders: one per criterion (code check, calibrated judge or human), with the rubric
  • Pass thresholds: per criterion and per failure type, plus the baseline to compare against
  • Regression trigger: which changes rerun the set, and what score drop blocks a release
  • Budgets and owner: cost per request, p95 latency, and who acts when a number slips
Download the one-page cheat sheet (PDF)

Leave at the door

Five myths this module breaks.

MythWhat’s actually true
“LLM-as-a-judge is objective.”It has biases. Calibrate it against human labels.
“I tried it a few times. It works.”A few tries are anecdotes. Use a golden set.
“One accuracy number tells the story.”Break failures down by type and by cost.
“The best model belongs in production.”The best fit for quality, cost and latency does.
“Evals are a launch step.”Rerun them on every prompt, model or data change.