AI security glossary
Key terms from this course, in plain English.
- Adversarial example
- An input crafted, often imperceptibly, to make a model produce a wrong output.
- AI bill of materials
- A record of the models, datasets, libraries and services an AI system depends on.
- AI red teaming
- Deliberately attacking an AI system to find security and safety weaknesses.
- Backdoor
- A hidden behaviour implanted in a model, usually through poisoning, that activates on a trigger.
- Data poisoning
- Manipulating training, fine-tuning or retrieval data to change a model's behaviour.
- Defence in depth
- Layering multiple controls so that if one fails, others still protect the system.
- Denial of wallet
- An attack that drives up the cost of an AI service through excessive usage.
- Direct prompt injection
- An attacker typing instructions designed to override a model's intended behaviour.
- Excessive agency
- OWASP LLM06: an AI system with more functionality, permissions or autonomy than needed.
- Improper output handling
- OWASP LLM05: passing model output to other systems without validation.
- Indirect prompt injection
- Malicious instructions hidden in content an AI processes, such as web pages, emails or documents.
- Jailbreak
- A prompt designed to bypass a model's safety training.
- Least privilege
- Giving each component only the minimum access it needs.
- Membership inference
- Determining whether a specific record was in a model's training data.
- MITRE ATLAS
- A knowledge base of adversary tactics and techniques against AI systems.
- Model Context Protocol (MCP)
- An open protocol for connecting AI models to tools and data sources.
- Model extraction
- Copying a model's behaviour by querying it and training a replica on the outputs.
- Model inversion
- Reconstructing sensitive features of training data from a model's outputs.
- OWASP Top 10 for LLM Applications
- The OWASP GenAI Security Project's list of the most critical LLM application risks (2025 edition).
- PyRIT
- Microsoft's open-source Python Risk Identification Tool for automating generative AI red teaming.
- Safetensors
- A model file format designed to store weights safely, without executing code when loaded.
- SAIF
- Google's Secure AI Framework, with six core elements for securing AI.
- Sandboxing
- Running code or tools in an isolated environment to limit damage.
- System prompt leakage
- OWASP LLM07: exposure of system prompt contents, which should never contain secrets.
- Tool poisoning
- Hidden instructions in a tool's description or responses that manipulate the model.
- Unbounded consumption
- OWASP LLM10: uncontrolled resource usage leading to cost, denial of service or extraction.