Fine-tuning methods
There are many ways to adapt a model. Knowing what each changes — and what it needs — helps you pick the cheapest one that works.
Learning objectives
- Contrast full fine-tuning and PEFT
- Explain how LoRA and QLoRA work
- Match SFT, DPO, RFT, continued pre-training and distillation to use cases
- Choose a suitable base model
2.1 Full fine-tuning
Update all of the model's parameters. Maximum flexibility, but for modern LLMs it needs a lot of GPU memory, produces a full-size copy of the model for every tune, and risks catastrophic forgetting — losing general abilities as the model specialises.
2.2 Parameter-efficient fine-tuning (PEFT)
PEFT methods freeze the original weights and train a small number of extra parameters. The most popular is LoRA (Low-Rank Adaptation): alongside a frozen weight matrix, it learns two small matrices whose product is a low-rank "update". Typically well under 1% of the parameters are trained.
- Much less memory and faster training.
- Tiny adapter files (megabytes, not gigabytes) — you can keep many adapters for one base model and swap them.
- Less forgetting, because the base weights are untouched.
- QLoRA goes further by loading the frozen base model in 4-bit precision, making it possible to tune fairly large models on a single GPU.
2.3 What you're optimising for: training objectives
| Method | Training data | Good for |
|---|---|---|
| Supervised fine-tuning (SFT) | Examples of inputs with ideal outputs | Formats, classification, extraction, style, instruction following |
| Preference tuning — e.g. Direct Preference Optimization (DPO) | For each prompt, a preferred and a rejected response | Tone, helpfulness, what to emphasise in summaries |
| Reinforcement fine-tuning (RFT) | Tasks plus a grader that scores answers | Improving reasoning on tasks with checkable answers |
| Continued pre-training | Large amounts of unlabelled domain text | Specialist vocabulary (e.g. legal, clinical), before SFT |
| Distillation | Outputs from a larger "teacher" model | Teaching a small model to imitate a big one on a task (Module 6) |
These are the same ingredients model developers use in post-training (see How AI Works, Module 5) — applied to your narrow task.
2.4 Choosing a base model
- Start from an instruction-tuned model, not a raw base model, for most tasks
- Prefer the smallest model that reaches your quality bar after tuning
- Check the licence permits your commercial use and fine-tuning
- Check language and domain coverage
- Consider where it will be served and at what cost
Key takeaways
- Full fine-tuning updates everything: costly and risks forgetting.
- LoRA trains small adapter matrices while the base stays frozen; QLoRA adds 4-bit loading.
- SFT learns from ideal outputs; DPO from preferred vs rejected; RFT from a grader.
- Start from the smallest suitable instruction-tuned model with a permissive licence.