Module 1 of 6 · 25 minutes

From prototype to production

A demo takes an afternoon; production takes discipline. This module maps what it takes to run generative AI reliably.

Learning objectives

  • Identify what changes between prototype and production
  • Compare LLMOps with classic MLOps
  • Describe the components of a production LLM application
  • Outline the LLMOps lifecycle

1.1 The prototype trap

A generative AI demo can be built in an afternoon. Getting it to work reliably for thousands of users — accurately, safely, affordably and fast — takes most of the effort. Common surprises when moving to production:

  • Quality that looked great on ten examples falls apart on real, messy inputs.
  • Costs grow with every user and every token.
  • Latency is too slow for the experience you designed.
  • The provider updates or retires a model and behaviour changes.
  • Users find inputs — and attacks — you never imagined.

1.2 LLMOps and MLOps

MLOps is the set of practices for reliably building, deploying and maintaining machine learning models: versioning, automated testing, deployment pipelines and monitoring. LLMOps applies those ideas to applications built on large language models, with different emphases:

Classic MLOpsLLMOps
What you change mostTraining data and modelsPrompts, retrieval, tools, model choice
EvaluationClear metrics (accuracy, RMSE)Open-ended outputs: rubrics, LLM judges, human review
Main costsTraining computePer-token inference
Main risksDrift, biasHallucination, prompt injection, data leakage, vendor model changes

1.3 Anatomy of a production LLM application

Behind a simple chat box, a production system typically has:

  • An application layer — authentication, user interface, business logic.
  • An AI gateway — one place to call models, handle keys, retries, rate limits, routing and logging.
  • Versioned prompts and configuration — stored and deployed like code.
  • Retrieval and tools — knowledge bases, APIs, permissions.
  • Guardrails — input and output checks (Module 3).
  • Observability — traces, metrics, feedback (Module 5).
  • An evaluation suite — run on every change (Module 2).
UsersApplicationInput guardrailsAI gatewaykeys · retries · routingOutput guardrailsModel AModel B (fallback)Retrieval & toolsObservability& evalsVersioned prompts & config feed every layer
A typical production architecture. Guardrails wrap the model, a gateway handles every call, and everything is observed.

1.4 The LLMOps lifecycle

  1. Design the use case, success criteria and risks.
  2. Build prompts, retrieval, tools — with an evaluation set from day one.
  3. Evaluate offline against criteria and baselines.
  4. Deploy gradually, with guardrails and rollback.
  5. Monitor quality, safety, cost and latency in production.
  6. Improve — turn failures into test cases and iterate.
DesignBuildEvaluateDeployMonitorImproveLLMOpslifecycle
The LLMOps lifecycle. Production failures become evaluation cases, closing the loop.

Key takeaways

  • Quality, cost, latency, model changes and adversarial users surprise teams in production.
  • LLMOps emphasises prompts, retrieval, open-ended evaluation and per-token cost.
  • Production apps need a gateway, versioned prompts, guardrails, observability and evals.
  • Design → build → evaluate → deploy → monitor → improve.