Metrics and experimentation for AI
AI features can look successful while quietly failing. This module shows how to measure what matters and run experiments you can trust.
Learning objectives
- Build a five-layer metrics tree for an AI feature
- Combine offline and online evaluation
- Design experiments that handle noise, novelty and automation bias
- Report AI value honestly
4.1 A metrics tree for AI products
AI features need more than one number. A metrics tree connects the business outcome to the signals underneath it, so a rise in usage can't hide a fall in quality, and a quality gain can't hide a cost blow-out.
| Layer | Example — AI support assistant |
|---|---|
| Business outcome | Cost per resolved case; customer satisfaction |
| User outcome | Cases resolved without a human; time to resolution |
| Adoption | Share of eligible cases where the assistant is used; repeat use |
| Quality | Correct and grounded answers (sampled review, LLM judge validated against humans) |
| Guardrails | Cost per case, latency, complaints, escalations, harmful outputs |
Google Cloud's guide to gen AI KPIs groups similar measures into model quality, system, adoption, operational and business-value metrics.
4.2 Offline and online evaluation
Offline (before release)
- Run the eval set on every change
- Fast, cheap and repeatable
- Only as good as the eval set — misses what real users do
Online (in production)
- A/B tests, sampled human review, LLM judges on live traffic, user behaviour
- Shows real-world value
- Slower, noisier, and exposes users to the change
You need both. Offline evals decide what is safe to test; online evidence decides what is worth keeping. OpenAI's evaluation best practices recommend datasets that mirror production and graders calibrated against human labels.
4.3 Experimenting with non-deterministic features
- Plan for noise — outputs vary, so you usually need larger samples or longer tests to detect a real difference
- Watch for novelty effects — usage of a shiny AI feature often spikes then settles; run tests long enough to see the plateau
- Measure downstream outcomes — resolved cases, retained customers, fewer corrections — not just clicks on the AI button
- Don't trust acceptance alone — users may accept wrong suggestions out of automation bias; pair acceptance rate with sampled quality checks
- Segment the results — new vs experienced users, languages, customer types; an average can hide a group that's worse off
- Keep a holdout — a small group without the feature lets you measure long-term value honestly
4.4 Proving value honestly
"AI saves 30 minutes a day" claims are common and often weak. Stronger evidence uses a baseline measured before launch, a comparison group, and outcome measures that matter to the business. Count the hidden costs too: time spent checking and fixing AI output, and the cost of errors that slip through. Report ranges and confidence, not a single heroic number.
4.5 Reporting to leadership
A good AI product review fits on one page: outcome trend against target, adoption, quality from sampled reviews, top failure types and what's being done, cost per task against budget, and any incidents. Leaders should see the bad news as clearly as the good.
For an AI feature, write one metric at each of the five layers in 4.1, the target for each, and how you'll measure it. Then design an experiment: what's compared, for how long, and which result would make you roll back.
See a worked example
Example — AI summary of long customer emails for agents: Business: handling cost per email (−10%). User: handling time (−20%). Adoption: summaries opened on 60% of eligible emails. Quality: 95% of sampled summaries rated accurate and complete. Guardrails: zero missed refund requests in sampled reviews; under 2s latency. Experiment: 50/50 split of agent teams for 6 weeks with a 10% holdout afterwards; roll back if missed-request rate rises.
Key takeaways
- Connect business outcomes to quality and guardrail metrics.
- Offline evals decide what's safe to test; online evidence decides what to keep.
- Run tests long enough to see past novelty.
- Acceptance rate alone can hide automation bias.
- Use baselines, comparison groups and ranges.