Preparing training data
A fine-tuned model is a mirror of its training data. This module shows how to build a dataset worth learning from.
Learning objectives
- Format examples in the JSONL chat format
- Assess data for correctness, consistency, coverage and lawfulness
- Use historical, expert and synthetic sources appropriately
- Validate, de-duplicate and split a dataset in Python
- Handle personal data and rights
3.1 Data is the product
The quality of a fine-tuned model is limited by the quality of its training examples. A few hundred excellent, consistent examples often beat many thousands of noisy ones. Every flaw — an inconsistent format, a wrong label, a rude reply — will be learned.
3.2 The chat format
Most tools expect training data as JSONL (one JSON object per line), each holding a short conversation in the same format the model will see in use:
train.jsonl · one example per line{"messages": [
{"role": "system", "content": "You write product descriptions for Good For You. Warm, concise, UK English."},
{"role": "user", "content": "Product: bamboo chopping board, large, FSC-certified, juice groove"},
{"role": "assistant", "content": "Meet your new kitchen favourite. Our large FSC-certified bamboo board ..."}
]}Keep the system prompt the same as you'll use in production, so the model learns the right context.
3.3 What good training data looks like
| Quality | What it means |
|---|---|
| Correct | Every output is one you'd be happy to ship |
| Consistent | Same format, tone and decisions for similar inputs |
| Representative | Covers the real range of inputs, including hard and unusual ones |
| Balanced | Classes and cases in sensible proportions |
| Clean | No duplicates, no leakage of test examples, no broken records |
| Lawful | You have the right to use it; personal data handled lawfully; licences respected |
3.4 Sources of examples
- Historical data — past tickets and replies, approved documents. Filter to the best.
- Expert-written examples — slow but high quality; ideal for edge cases.
- Synthetic data — generated by a strong model, then reviewed. Fast, but can introduce subtle errors and a monotonous style. Always check a sample, and check the provider's terms on using outputs for training.
3.5 Splits and checks
Python · validate and split a JSONL datasetimport json, random
rows = [json.loads(line) for line in open("train_all.jsonl")]
valid_roles = {"system", "user", "assistant"}
clean = []
for r in rows:
msgs = r.get("messages", [])
if (msgs and all(m.get("role") in valid_roles and m.get("content") for m in msgs)
and msgs[-1]["role"] == "assistant"):
clean.append(r)
# remove exact duplicates
seen, unique = set(), []
for r in clean:
key = json.dumps(r, sort_keys=True)
if key not in seen:
seen.add(key); unique.append(r)
random.seed(42); random.shuffle(unique)
n = len(unique)
splits = {"train": unique[:int(.8*n)], "val": unique[int(.8*n):int(.9*n)], "test": unique[int(.9*n):]}
for name, data in splits.items():
with open(f"{name}.jsonl", "w") as f:
f.writelines(json.dumps(r) + "\n" for r in data)
print(name, len(data))The test split is sacred: never train on it, never tune settings against it. Use the validation split to watch for overfitting during training.
3.6 Personal data and rights
Fine-tuned models can memorise training examples and reproduce them. Remove or pseudonymise personal data you don't need, respect data protection law (a DPIA is often appropriate), and make sure you have the rights to the content you train on.
Key takeaways
- A few hundred excellent examples can beat thousands of noisy ones.
- Match training examples to the production prompt format.
- Good data is correct, consistent, representative, balanced, clean and lawful.
- Review synthetic data; keep the test split untouched.
- Models can memorise training data — minimise personal data.