Guardrails and human-in-the-loop
An agent that can act can also act wrongly. This module shows how to give agents power without giving up control.
Learning objectives
- Identify agent failure modes
- Apply guardrails: least privilege, action tiers, limits, sandboxing and separation
- Implement human approval and business rules in code
- Design escalation and failure handling
- Recognise prompt injection as the main agent security risk
5.1 What can go wrong
- Wrong action: misreads the goal, calls the wrong tool, or uses wrong arguments.
- Runaway loops: repeats the same failing step, burning time and money.
- Compounding errors: a small early mistake propagates through later steps.
- Manipulation: prompt injection in emails, web pages or documents redirects the agent.
- Over-reach: takes an action beyond what the user intended — deleting, sending, paying.
5.2 Guardrails by design
| Guardrail | Implementation |
|---|---|
| Least privilege | Only the tools and data the task needs; read-only by default; scoped, short-lived credentials |
| Action tiers | Classify tools as safe / needs approval / forbidden |
| Human-in-the-loop | Pause for approval before consequential or irreversible actions |
| Input validation | Check every tool argument in code — types, ranges, allow-lists |
| Limits | Max steps, max spend, timeouts, rate limits |
| Sandboxing | Run code and browsing in isolated environments |
| Separation | Agents that read untrusted content don't also hold powerful tools |
| Logging | Record every step for audit and debugging |
5.3 Human approval in code
Python · approval gate for risky toolsNEEDS_APPROVAL = {"issue_refund", "send_email", "delete_record"}
REFUND_LIMIT = 50.00
def execute(call):
if call.name == "issue_refund" and float(call.input["amount"]) > REFUND_LIMIT:
return {"error": f"Refunds over £{REFUND_LIMIT:.0f} must be escalated to a manager."}
if call.name in NEEDS_APPROVAL:
print(f"Agent wants to run {call.name} with {call.input}")
if input("Approve? (y/n) ").strip().lower() != "y":
return {"error": "A human declined this action. Explain and suggest alternatives."}
return TOOL_FUNCS[call.name](**call.input)Note that business rules (the £50 limit) live in code, not just in the prompt — prompts can be ignored or manipulated; code can't.
5.4 Designing good escalation
Good agents know when to stop. Give them an explicit way to hand over ("escalate_to_human" as a tool), tell them when to use it — ambiguity, low confidence, policy exceptions, angry customers — and design the hand-over so the human gets a clear summary of what's been done.
5.5 Failure handling
- Return clear, actionable errors from tools so the agent can recover
- Retry transient failures in code (with back-off), not by letting the agent loop
- Detect repetition — the same tool with the same arguments several times — and stop
- Make actions idempotent where possible, so a repeated call doesn't double-charge
- Have a kill switch and a rollback plan
Prompt injection is the agent's biggest security risk
An agent that reads untrusted content and can take actions can be hijacked. Combine the guardrails above, and avoid giving one agent private data, untrusted input and external communication all at once. See Securing AI agents.
Key takeaways
- Agents fail by wrong actions, loops, compounding errors, manipulation and over-reach.
- Classify tools as safe, approval-needed or forbidden.
- Enforce business rules in code, not only in prompts.
- Give agents an explicit, well-designed way to escalate.
- Assume injection; limit what a hijacked agent could do.