Fairness and bias
Fairness is the most discussed responsible AI principle — and the hardest. This module shows why 'fair' has several meanings and how teams make principled choices.
Learning objectives
- Identify six sources of bias
- Explain why removing protected attributes isn't enough
- Compare demographic parity, equalised odds and calibration
- Explain why fairness definitions can conflict
- Describe how to measure and mitigate bias
3.1 Where bias comes from
| Source | What happens | Example |
|---|---|---|
| Historical bias | Data accurately reflects an unfair world | Past promotions favoured one group; a model learns to do the same |
| Representation bias | Some groups are under-represented in training data | Skin-condition images mostly of lighter skin |
| Measurement bias | The thing measured is a poor proxy for what matters | Using healthcare spending as a proxy for health need, when some groups historically received less care |
| Aggregation bias | One model for groups that differ in important ways | A single clinical threshold that works poorly for some populations |
| Evaluation bias | Test data doesn't reflect real users | Speech recognition benchmarked mainly on one accent |
| Deployment bias | A system is used in a context it wasn't designed for | A tool built for one country used in another |
3.2 Proxies: removing the column isn't enough
Simply deleting a protected characteristic such as ethnicity or gender from the data rarely removes bias. Other features — postcode, school, name, gaps in employment, hobbies — can act as proxies, and models are very good at finding them. Under the UK Equality Act 2010, a practice that disadvantages a protected group can be indirect discrimination unless it can be objectively justified.
3.3 What does "fair" mean? Competing definitions
There are many mathematical definitions of fairness. Three common ones, for a model that approves or rejects applications:
| Definition | Requires | Intuition |
|---|---|---|
| Demographic parity | Equal approval rates across groups | Outcomes should be shared equally |
| Equal opportunity / equalised odds | Equal true-positive (and false-positive) rates across groups | Equally qualified people should be treated alike; errors shouldn't fall more on one group |
| Calibration | A given risk score means the same thing for every group | "70% risk" should be 70% for everyone |
Research published in 2016–17 showed that when underlying rates differ between groups, it is generally mathematically impossible to satisfy calibration and equal error rates at the same time. This is exactly what the COMPAS debate turned on. Choosing a fairness definition is therefore a values decision that should be made deliberately, with affected stakeholders in mind — not left to a default setting.
3.4 Measuring fairness in practice
- Identify affected groups — including protected characteristics and intersections (e.g. older women, not just women or older people).
- Choose metrics that reflect the harm you're worried about — e.g. false rejections in lending.
- Disaggregate performance by group rather than relying on overall accuracy.
- Investigate gaps — are differences justified by legitimate factors?
- Document decisions and trade-offs.
Open-source toolkits such as Microsoft's Fairlearn and IBM's AI Fairness 360 support this analysis. Collecting demographic data for testing raises its own privacy questions, which need careful handling.
3.5 Mitigating bias
| Stage | Techniques |
|---|---|
| Before training | Collect more representative data; fix labels; reweight under-represented groups; question the target variable |
| During training | Add fairness constraints to the objective |
| After training | Adjust decision thresholds; add human review for borderline cases |
| In deployment | Monitor outcomes by group; give people ways to contest decisions; be ready to pause |
And sometimes the responsible answer is not to use AI for a particular decision at all.
3.6 Fairness in generative AI
For generative AI, fairness issues include stereotyped images and text, unequal quality across languages and dialects, and offensive content about particular groups. Testing uses counterfactual prompts (change only a name or attribute and compare outputs), red-teaming with diverse testers, and representative evaluation sets.
Key takeaways
- Bias enters through history, representation, measurement, aggregation, evaluation and deployment.
- Proxies mean deleting a column rarely removes bias.
- Fairness definitions can be mathematically incompatible — choosing one is a values decision.
- Disaggregate evaluation by group and intersection.
- Mitigate at every stage — or decide not to use AI.