Monitoring and observability

Once live, you need to see what your AI is doing — and know quickly when it goes wrong.

5.1 What to observe

5.2 Tracing

A trace follows one request through every step: the prompt version, retrieved documents, each model call (with tokens, latency and cost), tool calls, guardrail decisions and the final output. Traces turn "the bot said something weird" into a specific, fixable cause. OpenTelemetry and many LLM observability tools support this.

Python · structured logging of each model callimport json, logging, time, uuid
log = logging.getLogger("llm")

def traced_call(model_fn, prompt_id, prompt_version, user_id, **kwargs):
    trace_id, t0 = str(uuid.uuid4()), time.time()
    status, usage = "ok", {}
    try:
        result, usage = model_fn(**kwargs)          # return text and token usage
        return result
    except Exception as e:
        status = f"error:{type(e).__name__}"
        raise
    finally:
        log.info(json.dumps({
            "trace_id": trace_id, "prompt_id": prompt_id, "prompt_version": prompt_version,
            "user": user_id,                         # pseudonymous ID, not a name
            "latency_s": round(time.time() - t0, 3), "status": status, **usage,
        }))

5.3 Online quality monitoring

Sample a percentage of production traffic and score it with your automated graders and LLM judges, and send a smaller sample for human review. Track scores over time; a drop often signals a model update, a change in user behaviour or stale retrieval data.

5.4 Drift

Behaviour can change without any code change: providers update models, users ask new kinds of questions, documents go out of date. Watch the distribution of inputs (topics, languages, lengths) and outputs (refusal rate, length, sentiment) for shifts.

5.5 Alerts and incidents

Set alerts on thresholds — error rate, p95 latency, daily spend, guardrail triggers, quality score drops. Have an incident playbook: who's on call, how to switch to a fallback model or disable a feature, how to roll back a prompt, and how to notify users or regulators if needed. Turn every incident into new evaluation cases.

5.6 Privacy in logs

Logs of prompts and outputs can contain personal and confidential data. Minimise, pseudonymise or redact; restrict access; set retention periods; and record the lawful basis — the same data protection duties apply to logs as to the application.

Key takeaways

• Observe reliability, latency, cost, quality, safety and user signals.
• Traces connect a bad output to its cause.
• Score sampled production traffic automatically and with humans.
• Behaviour drifts without code changes — watch inputs and outputs.
• Logs carry personal data — minimise, protect and limit retention.