Module 2 of 6 · 35 minutes

Evaluating LLM quality

If you can't measure quality, every change is a guess. This module shows how to build evaluations that make shipping safe.

Learning objectives

Write specific, measurable success criteria
Build representative, versioned evaluation sets
Choose between code, LLM and human grading
Implement an evaluation harness and regression gate
Combine offline and online evaluation

2.1 Start with success criteria

Anthropic's guidance on evaluations recommends defining criteria that are specific, measurable, achievable and relevant before you build. Not "good answers", but, for example: "on a held-out set of 500 real customer questions, at least 90% of answers are rated correct and grounded, with no answers revealing personal data, and median latency under 3 seconds."

2.2 Build an evaluation set

Representative: mirrors real inputs — sample from logs where lawful
Edge cases: empty, very long, ambiguous, off-topic, multilingual, adversarial
Labelled: expected answers, or rubrics describing a good answer
Versioned: stored with the code; grows as you find new failures
Separate: a held-out set you don't tune against

2.3 Three ways to grade

Grader	Examples	Strengths	Weaknesses
Code	Exact match, valid JSON, contains required field, regex, length	Fast, cheap, reliable, scalable	Can't judge nuance
LLM judge	Rubric-based scores for correctness, tone, groundedness	Scales to open-ended outputs	Must be validated; can be biased
Human	Expert review, side-by-side preference	Highest quality and flexibility	Slow and expensive

Use code where you can, LLM judges where you must (checked against human samples), and humans for calibration and high-stakes decisions.

Use code graders wherever possible, validated LLM judges for nuance, and humans to calibrate both.

2.4 An evaluation harness

Python · a minimal evaluation harnessimport json, re, statistics, time

def grade_code(case, output):
    checks = {
        "valid_json": _is_json(output),
        "mentions_required": all(w.lower() in output.lower() for w in case.get("must_include", [])),
        "no_forbidden": not any(w.lower() in output.lower() for w in case.get("must_not_include", [])),
    }
    return all(checks.values()), checks

def _is_json(s):
    try: json.loads(s); return True
    except ValueError: return False

def run_eval(cases, generate, version):
    results, latencies = [], []
    for case in cases:
        t0 = time.time()
        out = generate(case["input"])
        latencies.append(time.time() - t0)
        ok, detail = grade_code(case, out)
        results.append({"id": case["id"], "pass": ok, **detail})
    pass_rate = sum(r["pass"] for r in results) / len(results)
    print(f"{version}: pass rate {pass_rate:.1%}, median latency {statistics.median(latencies):.2f}s")
    return pass_rate, results

2.5 Evals as a release gate

Run the evaluation suite automatically whenever prompts, retrieval, tools or models change — for example in your CI pipeline — and block releases that fall below thresholds or regress against the current version.

Python · fail the build on regressionbaseline, _ = run_eval(cases, generate_v12, "v12 (live)")
candidate, _ = run_eval(cases, generate_v13, "v13 (candidate)")
assert candidate >= 0.90, "Below the 90% quality bar"
assert candidate >= baseline - 0.01, "Regression against the live version"

Evaluations as a release gate in your CI pipeline: no change ships if it falls below the bar or regresses.

2.6 Offline vs online evaluation

Offline evaluation uses your test set before release. Online evaluation measures behaviour with real users: thumbs up/down, edits, escalations, task completion, A/B tests. You need both — offline to prevent regressions, online to discover what your test set missed.

Key takeaways

Define SMART-style success criteria first.
Evaluation sets need real inputs, edge cases, labels and versioning.
Prefer code graders; validate LLM judges against humans.
Run evals automatically and block regressions.
Offline evals prevent regressions; online signals reveal gaps.