Module 5 of 6 · 30 minutes

Building and launching

Shipping AI safely means iterating against evals, layering guardrails and rolling out in stages. This module covers the playbook.

Learning objectives

  • Run AI development in learning cycles
  • Apply the improvement ladder from prompts to fine-tuning
  • Plan guardrails
  • Design a staged rollout with a kill switch
  • Calculate cost per active user

5.1 Working with the build team

AI development is more experimental than traditional software. Plan in short learning cycles — try a prompt, retrieval change or model, run the evals, decide — rather than fixed-scope sprints. The PM keeps the team focused on the eval set and the user outcome, and makes the trade-off calls on quality, cost and latency.

5.2 The improvement ladder

When quality isn't good enough, try the cheapest fixes first:

  1. Better prompts — clearer instructions, examples, output format
  2. Better context — retrieve the right documents or data (RAG)
  3. Better workflow — split the task into steps, add checks, use tools
  4. Different model — larger for quality, smaller for speed and cost
  5. Fine-tuning — when you have many good examples and a consistent task

Each step should be justified by eval results, not intuition. See Fine-tuning for when step 5 is worth it.

Better promptsBetter context (RAG)Better workflowDifferent modelFine-tuningeffort, cost and time increase → try the lower rungs first
The improvement ladder. Climb only as far as your eval results require.

5.3 Guardrails

  • Input checks — block or route off-topic, abusive or prompt-injection attempts
  • Output checks — format validation, grounding, policy and safety filters
  • Action limits — permissions, spending caps, confirmation steps for agents
  • Rate and cost limits — per user and overall

The threats are covered in AI Security.

5.4 Launching safely

  • Internal dogfooding — staff use it first and report problems
  • Limited beta — a small group of real users, with close monitoring
  • Staged rollout — 5% → 25% → 100%, with a kill switch
  • A/B test — compare outcomes with and without the feature
  • Readiness review — evals passed, risk assessment done, support briefed, monitoring live
DogfoodstaffBetasmall groupStage 15%Stage 225%Everyone100%Kill switch readywatch evals, complaints, cost at each stage
A staged rollout widens exposure only as monitoring stays green, with a kill switch ready at every stage.

5.5 Unit economics

Know your cost per request and cost per active user. Heavy users of AI features can cost more than they pay, especially on flat-rate plans. Levers include caching repeated requests, routing simple requests to smaller models, limiting output length, and usage-based pricing or limits. Model the costs at your expected scale before you price the feature.

Activity: plan the rollout

Write a launch plan for your feature: stages, audience and duration of each, metrics watched at each stage, the conditions that trigger the kill switch, and the estimated cost per active user.

See a worked example

Example: Week 1–2 dogfood (finance team, 40 people). Week 3–6 beta (2 departments, 400 users) watching accuracy on a reviewed sample and edit rate. Then 25% → 100% over 4 weeks with an A/B comparison of time per claim. Kill switch if accuracy on sampled claims falls below 90% or any auto-submission occurs. Cost: ~£0.40 per active user per month.

Key takeaways

  • Iterate in short cycles judged by evals.
  • Prompts, context, workflow, model — then fine-tuning.
  • Guard inputs, outputs, actions and costs.
  • Dogfood, beta, stage, A/B test, with a kill switch.
  • Know cost per request and per active user.