Privacy, safety and human agency
Responsible AI protects people's data, keeps them safe and keeps them in control. This module covers those principles and AI's wider impacts.
Learning objectives
- Identify privacy risks at each stage of the AI lifecycle
- Apply data protection principles and DPIAs to AI
- Explain safety, robustness and fail-safe design
- Describe meaningful human agency and oversight
- Consider inclusiveness, environmental and societal impacts
5.1 Privacy
AI systems are hungry for data, and that creates privacy risks across the lifecycle:
- Collection: scraping or repurposing personal data people didn't expect to be used this way.
- Training: models can memorise and later reveal personal data from their training sets.
- Inference: AI can infer sensitive information — health, sexuality, political views — from seemingly harmless data.
- Use: staff pasting personal data into AI tools; chat logs stored and reviewed.
Data protection law (UK GDPR, EU GDPR) applies throughout. Key principles include lawfulness, fairness and transparency; purpose limitation; data minimisation; accuracy; storage limitation; and security. A Data Protection Impact Assessment (DPIA) is typically required for high-risk processing, which many AI uses involve.
Privacy-enhancing techniques include anonymisation and pseudonymisation, synthetic data, differential privacy and federated learning — each with trade-offs.
5.2 Safety and robustness
A safe AI system works reliably in the conditions it's used in, behaves sensibly in conditions it wasn't designed for, and resists misuse. Key ideas:
| Concept | Meaning |
|---|---|
| Reliability | Consistent performance in expected conditions |
| Robustness | Holding up under unusual, noisy or adversarial inputs |
| Fail-safe design | When it fails, it fails in a safe way — e.g. defers to a human |
| Misuse prevention | Guardrails against harmful use — e.g. generating dangerous instructions or abusive content |
| Monitoring | Detecting drift, incidents and new failure modes after launch |
For the most capable frontier models, developers have published safety frameworks — such as Anthropic's Responsible Scaling Policy and OpenAI's Preparedness Framework — that set capability thresholds requiring stronger safeguards. The AI Security & Safety course covers security in depth.
5.3 Human agency and oversight
Responsible AI should support human decision-making and autonomy, not quietly replace or manipulate it:
- Meaningful oversight — people with the competence, time and authority to intervene.
- No manipulation — AI shouldn't exploit psychological weaknesses. Manipulative AI causing significant harm is prohibited under the EU AI Act.
- Avoiding over-reliance — design that encourages users to think critically, not defer automatically.
- Choice — where appropriate, people can opt for a human alternative.
5.4 Inclusiveness and accessibility
Microsoft's principles call out inclusiveness: AI should empower everyone, including people with disabilities. AI can be a powerful accessibility tool (captions, screen readers, image descriptions) — and can exclude people when, for example, speech recognition fails on atypical speech. Include diverse users in design and testing.
5.5 Environmental and societal impact
- Environment: training and running large models uses significant energy and water for data centres. Choose appropriately sized models, and weigh benefits against costs.
- Work: AI changes tasks and jobs. Responsible adoption involves workers, invests in skills and considers who bears the costs.
- Information ecosystem: generative AI can spread misinformation and erode trust in genuine content. Labelling and provenance standards help.
- Creators: questions about training on copyrighted works, consent and compensation remain contested and are being addressed by courts and lawmakers.
Key takeaways
- AI creates privacy risks in collection, training, inference and use.
- Data protection principles and DPIAs apply to AI.
- Safe systems are reliable, robust, fail safely and resist misuse.
- Frontier developers publish safety frameworks with capability thresholds.
- AI should support human autonomy, include everyone and weigh environmental and social costs.