Documenting and governing data

Good data work leaves evidence. This module covers the documentation, protection and governance that make datasets trustworthy — and legally defensible.

6.1 Document your data: datasheets

A datasheet for a dataset records what anyone using it needs to know. Key sections:

• Motivation: why it was created and by whom
• Composition: what each record represents, size, fields, known gaps
• Collection: sources, time period, consent and legal basis
• Preparation: cleaning and labelling steps (your decision log)
• Uses: intended uses — and uses it is not suitable for
• Maintenance: owner, update schedule, version history

6.2 Lineage and versioning

Data lineage records where data came from and every transformation applied. Versioning means you can always say exactly which data trained a given model — essential for debugging, audits and reproducing results. Save cleaning code (not just cleaned files), and give each dataset release a version and date.

6.3 Data protection

If your data relates to identifiable people, UK GDPR (or EU GDPR) applies throughout the AI lifecycle:

Special category data (such as health or ethnicity) needs extra conditions; a Data Protection Impact Assessment is often required. The ICO's guidance on AI and data protection is the UK reference point.

6.4 Pseudonymisation in code

Python · replace IDs with salted hashesimport hashlib, os

SALT = os.environ.get("ID_SALT", "set-a-secret-salt")     # keep the salt secret and separate

def pseudonymise(value):
    return hashlib.sha256((SALT + str(value)).encode()).hexdigest()[:16]

clean["customer_key"] = clean["customer_id"].map(pseudonymise)
model_data = clean.drop(columns=["customer_id"])

Pseudonymised data is still personal data under GDPR, because it can be re-linked with the key — but it reduces risk.

6.5 The EU AI Act: data governance for high-risk AI

For high-risk AI systems, Article 10 of the EU AI Act requires training, validation and testing data to be subject to appropriate data governance and management practices, and to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose — with attention to possible biases. The practices in this course — profiling, cleaning logs, representativeness checks, documentation — are how organisations evidence that. See EU AI Act Fundamentals, Module 4.

6.6 Data quality is continuous

Data keeps changing after a model is deployed. Automate checks in data pipelines (types, ranges, allowed values, missing rates, row counts), alert on failures, and monitor for drift — shifts in the data your model sees compared with what it was trained on.

Python · simple automated checks for a pipelinedef check(df):
    problems = []
    if df["customer_key"].duplicated().any():          problems.append("duplicate customers")
    if not df["age"].between(16, 110).all():           problems.append("age out of range")
    if df["monthly_spend"].isna().mean() > 0.05:       problems.append("too many missing spend values")
    if not set(df["country"].dropna()) <= {"UK", "IE"}: problems.append("unexpected country code")
    return problems

print(check(model_data) or "All checks passed")

Key takeaways

• Datasheets record motivation, composition, collection, preparation, uses and maintenance.
• Version data and save cleaning code to make models reproducible.
• UK GDPR applies throughout the AI lifecycle; minimise and pseudonymise.
• Article 10 requires governed, representative, error-checked data for high-risk AI.
• Automate checks and monitor drift.