Red teaming, evaluation and secure operations

Security isn't a one-off review. This module covers how to test AI systems adversarially, evaluate them continuously and run them securely.

6.1 AI red teaming

Red teaming means deliberately attacking your own AI system to find weaknesses before others do — security vulnerabilities (injection, data leakage) and safety failures (harmful content, bias, dangerous capabilities). Microsoft's AI Red Team, drawing on its experience of red-teaming over 100 generative AI products, highlights lessons including:

• Understand what the system can do and where it's used — real-world impact matters more than clever attacks.
• Simple techniques often work; you don't need sophisticated attacks to find serious problems.
• Automation scales testing (Microsoft's open-source PyRIT tool, for example), but human judgement remains essential.
• Responsible AI harms are pervasive and hard to measure; red teaming is not the same as benchmarking.
• The work is never finished — systems and attacks keep changing.

6.2 Running a red-team exercise

1. Scope: which system, which risks (security, safety, fairness), which threat actors?
2. Team: security testers plus domain experts and diverse perspectives.
3. Test: direct and indirect injection, jailbreaks, data extraction, harmful content, excessive agency, abuse of tools, cost exhaustion.
4. Record: inputs, outputs, severity and reproducibility.
5. Fix and re-test; turn findings into automated regression tests.

6.3 Safety evaluation

Beyond attacks, systems need ongoing evaluation: accuracy and hallucination rates on realistic tasks, harmful-content rates, fairness across groups and robustness to unusual inputs. Build evaluation sets from real use (with privacy safeguards), automate them, and run them before every release and model change.

6.4 Frontier AI safety

For the most capable models, developers and governments focus on severe risks — such as misuse to help with chemical, biological, radiological or nuclear weapons, advanced cyber-offence, or models acting autonomously against their developers' intentions. Approaches include:

• Developer safety frameworks — Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework — which set capability thresholds that trigger stronger safeguards.
• Pre-deployment testing by government bodies such as the UK AI Security Institute.
• System cards publishing evaluation results with each major model release.
• The International AI Safety Report, an independent scientific assessment of advanced AI's capabilities and risks, with its 2026 edition published in February.
• Regulation: the EU AI Act requires providers of general-purpose AI models with systemic risk to evaluate models, run adversarial testing, report serious incidents and ensure cybersecurity.

6.5 Secure operations

The NCSC guidelines' fourth stage — secure operation and maintenance — applies directly:

• Logging and monitoring of prompts, outputs, tool calls and anomalies — within data protection rules
• Update management — track model and dependency versions; re-test after changes
• Incident response — AI-specific playbooks: disable tools, roll back models, purge poisoned data, rotate credentials, notify affected parties
• Information sharing — report vulnerabilities to vendors; learn from community resources such as MITRE ATLAS case studies
• Secure by default — safe configurations out of the box

6.6 AI for defenders

AI also strengthens security teams: triaging alerts, summarising incidents, reviewing code for vulnerabilities, and analysing malware. Google's SAIF encourages using automation to keep pace with threats. Apply the same governance and validation to security AI as to any other.

6.7 Putting it together: an AI security checklist

1. Know your scope (consumer app → self-trained model) and your AI bill of materials
2. Threat-model every system: untrusted inputs × sensitive data × possible actions
3. Defend against prompt injection in depth; assume some attacks succeed
4. Treat model output as untrusted; enforce access control outside the model
5. Apply least privilege and human approval to agents and tools
6. Secure data and model supply chains; prefer safe model formats
7. Set rate limits, quotas and budget alerts
8. Red-team before launch and regularly after; automate regression tests
9. Monitor, log and have an AI incident playbook
10. Align with OWASP, NCSC/UK Code of Practice, SAIF and NIST

Key takeaways

• Red-team for security and safety; simple attacks often work; automate but keep humans.
• Turn findings into regression tests.
• Evaluate before every release and model change.
• Frontier developers use capability thresholds; governments test models; the EU AI Act requires adversarial testing for systemic-risk GPAI.
• Monitor, log, update and rehearse AI incident response.