Making models smaller, faster and cheaper
At scale, every token costs money and time. This module covers the techniques that make AI affordable — many of which don't need fine-tuning at all.
Learning objectives
- Explain quantisation and estimate memory savings
- Describe distillation from teacher to student
- Use pruning and smaller models appropriately
- Apply routing, caching, batching and output limits
- Estimate and compare running costs
6.1 Why optimise?
At scale, the cost and speed of each model call dominate. A model that's 10× cheaper and 3× faster can change what's economically possible. Fine-tuning is one route; there are several others — often simpler.
6.2 Quantisation
Models store parameters as numbers with a certain precision — commonly 16 bits each. Quantisation stores them with fewer bits (8 or 4), cutting memory and often speeding up inference, usually with a small loss in quality. A 7-billion-parameter model needs roughly 14 GB of memory at 16-bit, about 7 GB at 8-bit and about 3.5 GB at 4-bit (before overheads) — the difference between needing a data-centre GPU and running on a laptop.
6.3 Distillation
In distillation, a large, capable "teacher" model generates high-quality outputs for many examples of your task, and a small "student" model is fine-tuned on them. The student learns to imitate the teacher on that narrow task at a fraction of the cost. Check the teacher's terms of use permit this.
6.4 Pruning and smaller models
Pruning removes weights or whole components that contribute little. In practice, the easiest win is often simply choosing a smaller model from the same family and testing whether it's good enough — perhaps with a better prompt or a light fine-tune.
6.5 System-level optimisations
| Technique | How it saves |
|---|---|
| Model routing | Send easy requests to a small model and hard ones to a large model |
| Prompt caching | Reuse the processed form of a long, repeated prompt prefix — supported by several providers to cut cost and latency |
| Shorter prompts | Fewer tokens per call; fine-tuning can replace long instructions and examples |
| Batching | Process non-urgent requests in bulk, often at a discount |
| Response caching | Store answers to identical requests |
| Output limits | Cap maximum output length; ask for concise formats |
6.6 Do the maths
Python · rough monthly cost comparisondef monthly_cost(requests_per_day, in_tokens, out_tokens, price_in, price_out):
# prices are per million tokens
per_call = in_tokens / 1e6 * price_in + out_tokens / 1e6 * price_out
return requests_per_day * 30 * per_call
# Illustrative US-dollar prices per million tokens — check your provider's current price list
large = monthly_cost(50_000, in_tokens=1_500, out_tokens=200, price_in=3.00, price_out=15.00)
small = monthly_cost(50_000, in_tokens=300, out_tokens=200, price_in=0.25, price_out=1.25)
print(f"Large model, long prompt: ${large:,.0f}/month")
print(f"Small tuned model, short prompt: ${small:,.0f}/month")Include the one-off costs (data, training, evaluation) and ongoing costs (hosting, re-tuning) to find the break-even point.
6.7 A decision checklist
- Is the task well defined, with a test set and success criteria?
- Have prompting, a better model and RAG been tried?
- Do you have enough high-quality, lawful examples?
- Does the tuned model beat baselines on quality, cost or speed?
- Have general ability, safety and memorisation been re-checked?
- Is there an owner and a plan for when the base model changes?
Key takeaways
- Quantisation stores weights in fewer bits — roughly 16-bit → 4-bit cuts memory by about 4×.
- Distillation teaches a small model to imitate a large one on a task.
- Often the simplest win is a smaller model with a better prompt.
- Routing, prompt caching, batching and output limits cut cost at system level.
- Do the maths, including one-off and ongoing costs.