Labelling and annotation
For supervised learning, labels are the lesson. This module shows how to make them consistent, efficient and trustworthy.
Learning objectives
- Explain how label quality limits model quality
- Write effective labelling guidelines
- Measure inter-annotator agreement with Cohen's kappa
- Choose efficient labelling techniques, including LLM assistance
- Detect label errors and treat labellers responsibly
4.1 Why labels matter
Supervised learning learns from labels — the "right answers" attached to examples: is this email spam, which category is this ticket, is this X-ray abnormal? Inconsistent or wrong labels put a ceiling on model quality, and evaluation against wrong labels gives misleading scores.
4.2 Write labelling guidelines
- A clear definition of every label, with examples and counter-examples
- Rules for ambiguous and borderline cases — and an "unsure" option
- What to do when information is missing
- Worked examples reviewed by experts
- A version number — update the guidelines as you learn
4.3 Measure agreement between labellers
Have two or more people label the same sample independently and measure how often they agree. Cohen's kappa measures agreement between two labellers while correcting for agreement expected by chance: 1 is perfect, 0 is no better than chance. Low agreement usually means the guidelines — not the labellers — need work.
Python · inter-annotator agreementfrom sklearn.metrics import cohen_kappa_score
annotator_a = ["billing", "delivery", "product", "delivery", "other", "billing", "product", "delivery"]
annotator_b = ["billing", "delivery", "product", "billing", "other", "billing", "delivery", "delivery"]
agreement = sum(a == b for a, b in zip(annotator_a, annotator_b)) / len(annotator_a)
kappa = cohen_kappa_score(annotator_a, annotator_b)
print(f"Raw agreement: {agreement:.0%} Cohen's kappa: {kappa:.2f}")
# Which items do they disagree on? Discuss them and refine the guidelines.
print([i for i, (a, b) in enumerate(zip(annotator_a, annotator_b)) if a != b])4.4 Making labelling efficient
| Technique | How it helps | Watch out for |
|---|---|---|
| Active learning | The model picks the examples it's least sure about for humans to label next | Can under-sample easy-but-important cases |
| Weak supervision | Simple rules or keywords label data roughly at scale | Rules carry their own biases and errors |
| LLM-assisted labelling | A language model proposes labels; humans review | Plausible but wrong labels; check a sample and measure agreement with humans |
| Consensus labelling | Several labellers per item; take the majority | More cost per item |
4.5 Finding label errors
Even well-run projects contain label errors. Useful checks: examples where a trained model confidently disagrees with the label; duplicates or near-duplicates with different labels; and spot audits of random samples by an expert.
Python · conflicting labels on duplicate texttickets = pd.DataFrame({
"text": ["Where is my order?", "Where is my order?", "Charged twice", "Refund please"],
"label": ["delivery", "billing", "billing", "billing"],
})
conflicts = tickets.groupby("text")["label"].nunique()
print(conflicts[conflicts > 1]) # same text, different labels -> review4.6 Labellers are people
Much labelling work is outsourced. Responsible practice includes fair pay, clear instructions, reasonable workloads, and protection and support for people asked to review distressing content.
Key takeaways
- Wrong labels cap model quality and distort evaluation.
- Guidelines need definitions, examples, edge-case rules and versions.
- Low agreement usually means unclear guidelines.
- LLM-proposed labels need human review and measured agreement.
- Look for conflicts, confident model disagreements and audit samples.