The AI threat landscape
AI systems inherit every traditional security risk — and add new ones. This module maps the landscape and the frameworks defenders use.
Learning objectives
- Explain what makes AI systems distinctively vulnerable
- Distinguish security of AI, AI for security and AI-enabled threats
- Distinguish AI security from AI safety
- Identify the main AI security frameworks
- Use the AWS scoping matrix to define responsibilities
1.1 Why AI security is different
AI systems are software, so all the usual security basics still apply: access control, patching, secure configuration, logging. But AI adds new properties that attackers exploit:
- Instructions and data are mixed. In an LLM, the developer's instructions and untrusted content (emails, web pages, documents) are all just text. The model can't reliably tell them apart.
- Behaviour is learned, not coded. You can't read the source code to see what a model will do; it can be influenced through its training data.
- Outputs are unpredictable. The same input can produce different outputs, and small changes can have large effects.
- AI increasingly acts. Agents with tools can send emails, move money or change systems, so a manipulated model can cause real-world damage.
- Complex supply chains. Pre-trained models, datasets, libraries, plugins and APIs from many parties.
1.2 Three relationships between AI and security
| Relationship | Examples | Covered |
|---|---|---|
| Security of AI — AI as a target | Prompt injection, data poisoning, model theft | This course |
| AI for security — AI as a defender's tool | Threat detection, triage, code review | Briefly, Module 6 |
| AI-enabled threats — AI as an attacker's tool | Deepfake fraud, convincing phishing at scale, faster vulnerability discovery | Briefly, below |
On AI-enabled threats, the International AI Safety Report 2026 and national cyber agencies note that AI is lowering the cost of some attacks — for example, highly personalised phishing and voice cloning. Basic controls (multi-factor authentication, verification of payment requests, patching) matter more than ever.
1.3 Safety and security
The terms overlap. Broadly, AI security is about protecting AI systems from deliberate attack and misuse; AI safety is about preventing AI from causing harm — including through errors, unintended behaviour and, for the most capable models, dangerous capabilities. This course covers both, with an emphasis on practical security.
1.4 The frameworks you'll use
| Framework | From | What it gives you |
|---|---|---|
| OWASP Top 10 for LLM Applications (2025) | OWASP GenAI Security Project | The ten most critical risks for LLM apps — the common language for AppSec teams |
| MITRE ATLAS | MITRE | A knowledge base of adversary tactics and techniques against AI systems, modelled on MITRE ATT&CK |
| Guidelines for secure AI system development | UK NCSC with US CISA and international partners | Security across four stages: secure design, development, deployment, and operation and maintenance |
| AI Cyber Security Code of Practice (January 2025) | UK Government | Baseline security principles across the AI lifecycle, feeding into a global ETSI standard |
| Secure AI Framework (SAIF) | Six core elements, from extending security foundations to AI to contextualising AI risk in business processes | |
| Generative AI Security Scoping Matrix | AWS | Five scopes, from consumer apps to self-trained models, to work out which controls you own |
| NIST AI RMF and GenAI Profile | NIST | Risk management, including information security risks of generative AI |
1.5 Know your scope
AWS's scoping matrix is a useful starting point. Your security responsibilities depend on how much of the AI stack you own:
- Consumer app — staff using a public AI service. Focus: acceptable use, data handling, account security.
- Enterprise app — a third-party business app with AI features. Focus: contracts, configuration, access.
- Pre-trained models — your app built on a foundation model API. Focus: application security, prompt injection, output handling.
- Fine-tuned models — a foundation model tuned on your data. Add: training data security and poisoning.
- Self-trained models — you own everything. Add: the full model supply chain and infrastructure.
Key takeaways
- LLMs mix instructions and data, learn behaviour from data, and increasingly act.
- AI is a target, a defensive tool and an attacker's tool.
- OWASP, MITRE ATLAS, NCSC, the UK Code of Practice, SAIF and NIST give you a shared language.
- Your controls depend on your scope — from consumer app to self-trained model.