Module 5 of 6 · 30 minutes

Evaluating fine-tuned models

A fine-tune is an experiment. This module shows how to prove it worked — and that it didn't break anything else.

Learning objectives

Compare a tuned model against prompt-only baselines
Choose task-appropriate metrics
Run a comparison on a held-out test set
Check for regressions in ability, safety, fairness and memorisation
Use blind review, A/B tests and model cards

5.1 Compare against baselines

A fine-tuned model is only worth it if it beats the alternatives on your task. Evaluate, on the same held-out test set:

The base model with your best prompt (no tuning)
A stronger general model with your best prompt
Your fine-tuned model

If the fine-tuned model doesn't clearly win on quality, cost or speed, don't ship it.

Illustrative results on a held-out test set. A fine-tune is worth shipping only if it clearly beats the baselines on quality, cost or speed.

5.2 Choose task metrics

Task type	Metrics
Classification	Accuracy, precision, recall, F1 per class
Extraction	Field-level exact match; valid-format rate
Generation (style, descriptions)	Rubric scores from human reviewers or a validated LLM judge; format compliance
Efficiency	Latency, tokens per call, cost per 1,000 requests

Python · compare models on a held-out test setimport json
from sklearn.metrics import classification_report

test = [json.loads(l) for l in open("test.jsonl")]
inputs = [t["messages"][-2]["content"] for t in test]      # last user turn
gold   = [t["messages"][-1]["content"] for t in test]      # expected label

for name, predict in {"base + prompt": base_predict, "fine-tuned": tuned_predict}.items():
    preds = [predict(x) for x in inputs]
    print(f"\n=== {name} ===")
    print(classification_report(gold, preds, zero_division=0))

5.3 Check for regressions

General ability: a small set of everyday tasks outside your domain — has it got worse?
Safety: red-team prompts the base model refused — does the tuned model still refuse?
Fairness: performance across relevant groups and languages
Memorisation: does it reproduce training examples verbatim, including personal data?
Robustness: unusual inputs, typos, very long inputs

5.4 Human review and A/B testing

For subjective outputs, have reviewers compare outputs blind (without knowing which model produced which). Before full roll-out, run an A/B test on a slice of real traffic and compare business metrics — such as edits needed, resolution rate or complaints.

5.5 Document it

Record the base model and version, data sources and size, training settings, evaluation results against baselines, known limitations and the owner — a lightweight model card. You'll need it when the base model changes, when auditors ask, or when something goes wrong.

Key takeaways

Always compare against base-model-plus-prompt and a stronger model.
Pick metrics that match the task — and include cost and latency.
Check general ability, safety, fairness, memorisation and robustness.
Use blind human review and A/B tests for subjective outputs.
Document it in a model card.