Specifying quality with evals
For AI features, the evaluation set is the specification. This module shows product managers how to define, measure and agree 'good enough'.
Learning objectives
- Explain why evals act as AI specifications
- Build a representative evaluation set
- Choose grading methods
- Write launch criteria and success metrics
4.1 Evals are the new spec
For AI features, the most precise specification is an evaluation set: a collection of realistic inputs with expected outputs or clear grading criteria. It turns "make it good" into something the team can measure on every change. Anthropic's guidance on defining success and building evaluations is a good starting point.
4.2 Building an evaluation set
- Start with 50–200 real, representative inputs (anonymised)
- Include the hard cases: ambiguous inputs, edge cases, adversarial attempts, different user groups and languages
- Write expected outputs, or a rubric describing what makes an output acceptable
- Have domain experts agree the expectations — product, legal or support, not only engineers
- Keep growing it: every production failure becomes a new test case
4.3 How to grade
| Method | Good for | Watch out for |
|---|---|---|
| Exact or code-based checks | Categories, extracted fields, formats | Can't judge open-ended text |
| Human review | Nuance, tone, domain correctness | Slow, costly, reviewers disagree — use clear rubrics |
| LLM as judge | Scaling rubric-based grading | Must itself be validated against human judgements |
4.4 Acceptance criteria for AI
Write launch criteria as thresholds on the eval set, including must-never conditions:
≥ 90% of answers rated correct and complete · ≥ 98% grounded in cited articles · 0 answers inventing refund or legal policy · p95 response time under 4 seconds · average cost under 2p per answer · no significant quality gap between English and Spanish questions.
4.5 Success metrics after launch
Eval scores show the AI is good; product metrics show it's useful. Pair them: task success rate, time saved, acceptance or edit rate, deflection or resolution rate, retention of users who adopt the feature, satisfaction, and guardrail metrics (complaints, escalations, harmful outputs, cost per active user).
For your feature, write: ten example eval inputs (including three hard cases), how each will be graded, and your launch criteria with at least one must-never condition.
See a worked example
Example — expense categorisation: Inputs include a clear taxi receipt, a blurry hotel bill, a foreign-currency restaurant receipt, a mixed meal-and-alcohol bill, and a handwritten receipt. Grading: exact match on category; human check on 'unreadable' handling. Launch: ≥ 95% top-1 correct, ≥ 99% top-3, 0 auto-submitted claims, under 2 seconds.
Key takeaways
- An eval set turns 'make it good' into a measurable target.
- Include hard cases and agree expectations with experts.
- Use code checks, humans and validated LLM judges.
- Launch criteria need thresholds and must-nevers.
- Pair eval scores with product outcome metrics.