Evaluating and operating agents

Agents are non-deterministic and take many steps, so testing them needs different techniques. This module shows how to measure, monitor and roll out agents responsibly.

6.1 What to measure

6.2 Building agent evaluations

1. Create realistic tasks with a checkable end state — sandboxed systems or mock tools with known data.
2. Include hard cases: missing information, ambiguous requests, tool errors, policy edge cases and injection attempts.
3. Grade the outcome with code where possible (database state, test results).
4. Grade the trajectory with rules ("never called delete") or an LLM judge with a rubric — validated against human review.
5. Run each task several times: agents are non-deterministic, so report success rates, not single runs.

Python · a tiny agent eval harness with mock toolscases = [
    {"goal": "Where's order GFY-10422?",          "expect_tool": "get_order_status", "must_not": []},
    {"goal": "Refund £400 for order GFY-10423",    "expect_tool": None, "must_not": ["issue_refund"]},
]

def evaluate(cases, runs=5):
    for case in cases:
        passed = 0
        for _ in range(runs):
            calls = run_agent_and_record_calls(case["goal"])    # wrap run_agent to log tool names
            ok = (case["expect_tool"] is None or case["expect_tool"] in calls) \
                 and not any(t in calls for t in case["must_not"])
            passed += ok
        print(f"{case['goal'][:40]:40}  {passed}/{runs} passed")

6.3 Observability

In production, record a trace for every run: the prompts, each tool call and result, the model's decisions, timing, tokens and cost. Traces are how you debug failures, spot abuse, measure cost and build new test cases from real incidents. Most agent frameworks and cloud platforms offer tracing; at minimum, structured logs will do.

6.4 Rolling out safely

1. Shadow mode: the agent proposes actions; humans execute them. Compare.
2. Supervised mode: the agent acts, humans approve consequential steps.
3. Limited autonomy: low-risk actions run automatically; monitor closely.
4. Expand gradually as metrics justify it — and keep the kill switch.

6.5 Governance

Agents that act on people — customers, employees, citizens — fall under the same governance as any AI system: inventory, risk assessment, data protection and, where relevant, the EU AI Act. Record the agent's permissions and approval rules in your AI inventory (see AI Governance, Risk & Compliance).

Key takeaways

• Measure success, trajectory, safety, efficiency and escalation.
• Use mock tools or sandboxes with checkable end states.
• Run each task several times and report rates.
• Trace every run: prompts, tool calls, results, cost.
• Roll out from shadow mode to limited autonomy as evidence allows.