Risks and how to spot them
AI fails in recognisable ways. Once you know the patterns, you can catch most problems before they reach a customer, a colleague or a decision.
Learning objectives
- Explain what hallucinations are and why they happen
- Recognise bias in AI outputs
- Describe automation bias and how to counter it
- Protect yourself against deepfake fraud
- Apply the CHECK routine before relying on AI output
5.1 Hallucinations
A hallucination is AI output that sounds plausible but is false or made up — an invented statistic, a non-existent court case, a fake reference or a wrong product detail. IBM describes three kinds: factual (false information), contextual (ignoring or mixing up the material you supplied) and consistency (contradicting itself).
OpenAI's 2025 research on why language models hallucinate argues that standard training and evaluation reward guessing over admitting uncertainty — like a test where a lucky guess scores better than leaving a blank. Hallucinations can be reduced, but not eliminated.
Courts in several countries, including England and Wales, have dealt with legal submissions citing cases that don't exist, generated by AI and not checked. The lesson isn't "never use AI" — it's "never present AI output as fact without verifying it".
5.2 Bias and unfairness
AI learns from historical data, and history contains bias. A CV-screening model trained on past hiring decisions can learn to prefer the kinds of people who were hired before. Generative AI can produce stereotyped images or text — for example, always picturing a "CEO" as a man. Watch for outputs that treat groups of people differently without good reason.
5.3 Over-reliance (automation bias)
Automation bias is our tendency to trust a machine's suggestion over our own judgement — even when we have evidence it's wrong. It gets stronger when we're busy, when the AI is usually right, and when its output looks polished. The fix is deliberate: decide in advance what you will check, every time.
5.4 Deepfakes, scams and manipulation
Generative AI makes it cheap to create convincing fake voices, videos and messages. Fraudsters have used cloned voices and deepfake video calls to impersonate senior leaders and request urgent payments.
- Treat urgency plus secrecy plus payment as a red flag — whatever the voice or face
- Verify unusual requests through a separate, known channel (call back on a number you already have)
- Follow payment-approval processes with no exceptions "because the CEO asked"
5.5 Security risks in AI tools
AI tools that read emails, web pages or documents can be tricked by prompt injection: hidden instructions inside the content that try to make the AI do something you didn't ask, such as revealing data. It's ranked the number one risk in the OWASP Top 10 for LLM Applications. As a user, be cautious about letting AI tools act on untrusted content and approve actions with care. The AI Security & Safety course covers this in depth.
5.6 A quick risk check
Before you rely on an AI output, run through CHECK:
| Consequences | What happens if this is wrong — and who is affected? |
| Hallucination | Are there facts, figures, names, quotes or references I need to verify? |
| Equity | Could this treat any group unfairly or reinforce a stereotype? |
| Confidentiality | Did I share, or will I publish, anything I shouldn't? |
| Keep accountable | Am I comfortable putting my name to this? Do I need to disclose AI use? |
Key takeaways
- Hallucinations can be reduced, not eliminated — verify facts, figures and references.
- AI can reproduce historical bias and stereotypes.
- Decide in advance what you'll check, to counter automation bias.
- Verify unusual payment requests through a separate channel, whatever the voice or face.
- CHECK: Consequences, Hallucination, Equity, Confidentiality, Keep accountable.