Managing cost, change and governance

Models, prices and platforms keep changing. This module covers how to keep a production AI system stable, affordable and accountable over time.

6.1 Version everything

An LLM application's behaviour depends on many moving parts. Record the version of each, for every release:

• Model name and version (pin specific versions where providers offer them)
• Prompts and system instructions
• Retrieval configuration and index version
• Tool definitions and permissions
• Guardrail settings
• Evaluation set version and results

6.2 Releasing changes safely

6.3 Model upgrades and deprecations

Providers release new models frequently and retire old ones on published schedules. Platforms change too: in 2026, for example, OpenAI announced it was deprecating its hosted Evals platform and winding down its fine-tuning platform for new users. Plan for change:

• Track providers' deprecation notices and put retirement dates in your calendar.
• Re-run your full evaluation suite on any new model before switching.
• Keep your evals, prompts and data portable — in your own repository, not only inside a vendor's tool.
• Abstract model calls behind your gateway so switching is a configuration change.

6.4 Managing cost

Python · a simple daily budget guardDAILY_BUDGET = 200.00          # in your billing currency
spent_today = 0.0

def record_cost(input_tokens, output_tokens, price_in, price_out):
    global spent_today
    spent_today += input_tokens / 1e6 * price_in + output_tokens / 1e6 * price_out
    if spent_today > 0.8 * DAILY_BUDGET:
        alert(f"80% of daily AI budget used: {spent_today:.2f}")
    if spent_today > DAILY_BUDGET:
        enable_degraded_mode()   # e.g. route to a cheaper model, pause non-essential features

Other levers: per-user quotas, smaller models via routing, prompt and response caching, batch processing for non-urgent work, shorter prompts and output limits, and attributing cost to features so you know where money goes.

6.5 Operational governance

Production AI sits inside your organisation's governance: the system is in the AI inventory with an owner; risk and impact assessments are kept current; changes that alter risk trigger review; incidents are reported; and documentation is maintained (see AI Governance, Risk & Compliance). For high-risk uses under the EU AI Act, post-market monitoring and logging are legal requirements.

6.6 A production readiness checklist

• Success criteria and an evaluation suite that gates releases
• Guardrails on input, output and actions; safety test suite
• Gateway with timeouts, retries, fallbacks, rate limits and quotas
• Tracing, dashboards and alerts for reliability, latency, cost, quality and safety
• Versioned prompts, models, retrieval and tools; one-step rollback
• Plan for model deprecations; portable evals and prompts
• Data protection for prompts, outputs and logs
• Named owner, incident playbook and governance records

Key takeaways

• Record the version of every component for every release.
• Release gradually with a one-step rollback.
• Track deprecations; re-evaluate before switching; keep evals portable.
• Budgets, quotas, routing, caching and batching control cost.
• Production AI stays inside governance: owner, assessments, incidents, documentation.