Transformers and how LLMs are trained

How does a next-token predictor become a helpful assistant? This module follows a large language model from raw text to finished product.

5.1 The transformer

In 2017, Google researchers published a paper titled "Attention Is All You Need", introducing the transformer. It's the "T" in GPT (Generative Pre-trained Transformer) and the foundation of essentially every major LLM today.

Its key idea is attention: when processing each token, the model looks at all the other tokens in the context and weighs how relevant each one is. In "The trophy didn't fit in the suitcase because it was too big", attention helps the model work out that "it" refers to the trophy. Transformers also process tokens in parallel, which made it practical to train on vastly more data using GPUs.

5.2 Stage 1: pre-training

A base model is trained on a huge corpus — web pages, books, code, articles and other sources — to predict the next token. This stage uses enormous amounts of compute and produces a model with broad knowledge and language ability. But a base model isn't a helpful assistant: ask it a question and it might just continue with more questions, as if completing a web page.

5.3 Stage 2: post-training

To turn a base model into an assistant like ChatGPT, Claude or Gemini, developers apply further training:

• Instruction tuning (supervised fine-tuning): training on examples of good instructions and helpful responses.
• Reinforcement learning from human feedback (RLHF): people compare pairs of responses; a reward model learns their preferences; the LLM is then trained to produce responses the reward model rates highly.
• AI feedback and principles: Anthropic's "Constitutional AI" approach uses a written set of principles, with AI-generated feedback, to train the model towards helpful, honest and harmless behaviour. Anthropic now publishes the constitution it uses for Claude.
• Safety training: teaching models to refuse clearly harmful requests and to resist manipulation.

5.4 Reasoning models

Since 2024, developers have released reasoning models that are trained (largely with reinforcement learning) to work through problems step by step before answering — spending more computation "thinking" on harder questions. They tend to do better on maths, coding and multi-step analysis, at the cost of speed and price. OpenAI's prompting guidance compares them to a senior colleague you can give a goal to, while faster general models need more explicit instructions.

5.5 Multimodal models

Modern frontier models are multimodal. Images, audio and documents are converted into token-like representations the transformer can attend to alongside text. That's how you can upload a chart and ask what it shows, or photograph a whiteboard and get a summary.

5.6 From models to tools and agents

On its own, an LLM only produces text. Applications extend it with tools: web search, code execution, file access and connections to other systems (for example through the open Model Context Protocol). An LLM that decides which tools to call, in a loop, to achieve a goal is an agent. The model's knowledge is frozen at training time, but tools let it look things up and act.

Key takeaways

• Transformers (2017) use attention to weigh relevant context, and train efficiently at scale.
• Pre-training gives broad knowledge; post-training makes a helpful, safer assistant.
• Training on human approval can encourage sycophancy.
• Reasoning models spend more computation thinking step by step.
• Tools and agents extend a frozen model with search and actions.