Attacks on data and models
Attackers don't only target prompts. They target the data models learn from, the models themselves, and the supply chain that assembles them.
Learning objectives
- Explain data and model poisoning, including backdoors
- Describe model theft, extraction and system prompt leakage
- Describe training-data extraction, membership inference and model inversion
- Explain adversarial examples
- Manage AI supply-chain risks
3.1 Data poisoning
Data and model poisoning (OWASP LLM04:2025) means manipulating training, fine-tuning or retrieval data to change a model's behaviour — degrading performance, introducing bias, or planting a hidden backdoor that activates on a specific trigger.
Poisoning may need less data than you'd think. In 2025, research by Anthropic with the UK AI Security Institute and The Alan Turing Institute found that a small, roughly constant number of malicious documents — around 250 in their experiments — could implant a simple backdoor in language models across a range of sizes. Large models trained on scraped web data are therefore exposed to anyone who can publish content online.
- Know the provenance of training and fine-tuning data
- Validate, filter and version datasets; restrict who can change them
- Protect retrieval (RAG) sources — poisoning a knowledge base is often easier than poisoning a model
- Test models for unexpected behaviour on trigger-like inputs
3.2 Model extraction and theft
Trained models are valuable intellectual property. Attackers may:
- steal model files (weights) through conventional breaches — protect them like any crown-jewel asset;
- extract a model's behaviour by querying it many times and training a copy on the outputs ("distillation" without permission);
- extract the system prompt (OWASP LLM07:2025 System Prompt Leakage) — never put secrets, credentials or security logic in a system prompt.
3.3 Privacy attacks on models
| Attack | What the attacker learns |
|---|---|
| Training data extraction | Verbatim memorised content, such as personal data or secrets, from the training set |
| Membership inference | Whether a specific record was in the training data (e.g. a patient in a hospital dataset) |
| Model inversion | Reconstructs sensitive features of training data from model outputs |
Mitigations include minimising sensitive data in training, de-duplication, differential privacy, output filtering and rate limiting.
3.4 Evasion (adversarial examples)
For classic machine learning, adversarial examples are inputs crafted to fool a model — small changes, often invisible to people, that cause misclassification. Researchers have shown stickers on road signs that confuse image classifiers, and malware tweaked to evade ML-based detection. Defences include adversarial training, input validation and not relying on a single model for security decisions.
3.5 The AI supply chain
Supply chain risks (OWASP LLM03:2025) are growing as organisations assemble AI from many parts:
- Pre-trained models from public hubs may be tampered with or contain malicious code. Some model file formats (such as Python "pickle" files) can execute code when loaded — prefer safer formats such as safetensors and scan files.
- Datasets may be poisoned or have unclear licences.
- Libraries and packages — including fake packages with names that AI coding assistants hallucinate, which attackers can register.
- Plugins, tools and MCP servers that connect models to other systems (Module 5).
- Third-party AI APIs whose models change without notice.
Keep an AI bill of materials: which models, datasets, libraries and services each system depends on, with versions and sources.
Key takeaways
- Poisoning can need surprisingly few malicious documents — protect data provenance, including RAG sources.
- Protect model weights; never put secrets in system prompts.
- Models can leak training data; minimise sensitive data and filter outputs.
- Adversarial examples fool classic ML models.
- Keep an AI bill of materials; prefer safe model formats; verify packages and tools.