Neural networks and deep learning
Neural networks power nearly every AI breakthrough of the last decade. Here's how they work — and why even their builders can't fully explain them.
Learning objectives
- Describe an artificial neuron and how layers build up features
- Explain parameters and the role of scale
- Explain why neural networks are 'black boxes' and what interpretability research does
- Match common architectures to the data they handle
3.1 Artificial neurons
A neural network is a model loosely inspired by the brain. Its basic unit, an artificial neuron, does something very simple: it takes some numbers in, multiplies each by a weight (how important it is), adds them up, and passes the result through a simple function that decides how strongly to "fire".
On its own, one neuron is not clever. The power comes from connecting very many of them in layers.
3.2 Layers learn features
In an image-recognition network:
- early layers learn to detect simple patterns — edges, colours, textures;
- middle layers combine these into shapes — eyes, wheels, letters;
- later layers combine shapes into whole concepts — a face, a car, a word.
Nobody programs these features. They emerge during training because they help reduce the error. Deep learning simply means using networks with many layers.
3.3 Parameters and scale
Every connection weight is a parameter learned in training. Early image networks had millions of parameters; today's large language models have billions, and the largest are thought to have far more. Broadly, more parameters, more data and more compute have produced more capable models — though efficiency improvements mean smaller models keep getting better too.
3.4 Why neural networks are hard to explain
A rule-based system can point to the rule it used. A neural network's "reasoning" is spread across millions or billions of weights, none of which means much on its own. This is why they're often called black boxes.
The field of interpretability tries to look inside. In 2025, Anthropic published research tracing some of the internal steps of its Claude model. Among the findings: when writing rhyming poetry, the model appeared to plan the rhyming word ahead of time; and when asked how it had solved a sum, its explanation described the standard school method even though its internal process worked differently. In other words, a model's explanation of its own reasoning may not reflect what actually happened inside it.
When an AI tool "explains its reasoning", treat that explanation as another output to evaluate — useful, but not a guaranteed window into how the answer was produced.
3.5 Different architectures for different data
| Architecture | Good for |
|---|---|
| Convolutional neural networks (CNNs) | Images and video — they scan for local patterns |
| Recurrent neural networks (RNNs) | Sequences such as text and speech — largely replaced by transformers |
| Transformers | Language, and now images, audio, code and more — the basis of today's LLMs (Module 5) |
| Diffusion models | Generating images and video by gradually removing noise |
Key takeaways
- Neurons weight and combine inputs; layers learn increasingly complex features.
- Parameters are learned weights — today's LLMs have billions.
- A model's own explanation of its reasoning may not reflect its internal process.
- Transformers now dominate language and much else; diffusion models generate images.