Representativeness, splits and leakage

Clean data can still mislead if it's unrepresentative or leaks the answer. This module covers the checks that separate trustworthy models from impressive-looking ones.

5.1 Is your data representative?

A dataset can be clean and still give a misleading picture if it doesn't reflect the people and situations the model will meet. Compare the make-up of your data with the population the model will serve: regions, age groups, languages, device types, customer segments, time periods.

Python · compare data with the population you'll servedata_share = clean["country"].value_counts(normalize=True)
target_share = pd.Series({"UK": 0.70, "IE": 0.30})      # from business records
print(pd.DataFrame({"in_data": data_share, "target": target_share}).round(2))

If a group is under-represented, consider collecting more data, re-weighting, or at least measuring performance for that group separately — and documenting the limitation.

5.2 Watch for proxies and historical bias

Features like postcode, school or employment gaps can act as proxies for protected characteristics. Labels based on past human decisions can carry past unfairness. These are data problems before they are model problems — see Responsible AI, Module 3.

5.3 Splitting data correctly

Python · a stratified train/test splitfrom sklearn.model_selection import train_test_split

X = clean[["age", "monthly_spend", "country"]]
y = clean["churned"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)   # keeps the churn rate similar in both

• Stratify on the label (and sometimes key groups) so rare classes appear in both splits.
• Split by time when predicting the future: train on older data, test on newer — never the reverse.
• Split by entity (customer, patient) so the same person doesn't appear in both training and testing.

5.4 Data leakage

Leakage happens when information that wouldn't be available at prediction time sneaks into training — producing models that look brilliant in testing and fail in reality.

5.5 Preparing features: encoding and scaling inside a pipeline

Models need numbers. Categories are commonly one-hot encoded (a 0/1 column per category), and numeric features are often scaled to similar ranges. Wrapping these steps in a scikit-learn pipeline fits them on training data only — the standard defence against preprocessing leakage.

Python · leakage-safe preprocessing pipelinefrom sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer

numeric = ["age", "monthly_spend"]
categorical = ["country"]

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

preprocess.fit(X_train)                 # learns medians, scales and categories from TRAINING data only
X_train_ready = preprocess.transform(X_train)
X_test_ready = preprocess.transform(X_test)

You'll attach a model to the end of this pipeline in Machine Learning Fundamentals.

Key takeaways

• Representativeness is a data quality issue — measure it.
• Proxies and historical labels can encode unfairness.
• Stratify; split by time for forecasting; split by entity.
• Leakage produces great test scores and poor real-world performance.
• Pipelines fit preprocessing on training data only.