Better models: overfitting, ensembles and validation
How do you know a model will work on tomorrow's data? This module covers the tools practitioners use to build models that generalise.
Learning objectives
- Diagnose overfitting from training vs test scores
- Explain random forests and gradient boosting
- Compare models with k-fold cross-validation
- Tune hyperparameters with grid search
- Use pipelines to prevent leakage
4.1 Overfitting, again
A model that memorises its training data performs brilliantly on it and poorly on new data. Compare training and test scores: a big gap signals overfitting.
for depth in [2, 4, 8, 16, None]:
t = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
print(f"depth {str(depth):4} train {t.score(X_train, y_train):.3f} test {t.score(X_test, y_test):.3f}")Unlimited depth reaches near-perfect training accuracy while test accuracy drops — classic overfitting.
4.2 Ensembles: many models are better than one
| Model | Idea | Strengths |
|---|---|---|
| Random forest | Many trees, each trained on a random sample of rows and features; they vote | Robust, little tuning, resists overfitting |
| Gradient boosting | Trees built one after another, each correcting the previous ones' errors | Often the most accurate on tabular data |
For structured, tabular business data, tree ensembles are frequently the best-performing choice — often outperforming deep learning.
4.3 Cross-validation
A single train/test split can be lucky or unlucky. k-fold cross-validation splits the training data into k parts, trains on k−1 and validates on the remaining one, k times, and averages the scores. Use it to compare models and settings; keep the test set for the final check.
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
models = {
"logistic": LogisticRegression(max_iter=1000),
"tree (depth 4)": DecisionTreeClassifier(max_depth=4, random_state=0),
"random forest": RandomForestClassifier(n_estimators=300, random_state=0),
"gradient boosting": HistGradientBoostingClassifier(random_state=0),
}
for name, m in models.items():
scores = cross_val_score(m, X_train, y_train, cv=5, scoring="roc_auc")
print(f"{name:18} AUC {scores.mean():.3f} ± {scores.std():.3f}")On this dataset the hidden pattern is close to linear, so logistic regression may do as well as the ensembles — a useful reminder that complex isn't always better.
4.4 Hyperparameter tuning
Hyperparameters are settings you choose before training (tree depth, number of trees, learning rate). Search over them with cross-validation:
from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(RandomForestClassifier(random_state=0),
{"n_estimators": [100, 300], "max_depth": [4, 8, None], "min_samples_leaf": [1, 10]},
cv=5, scoring="roc_auc", n_jobs=-1)
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))
print("Test AUC:", round(roc_auc_score(y_test, grid.predict_proba(X_test)[:, 1]), 3))4.5 Pipelines keep it honest
Put preprocessing (imputation, scaling, encoding) and the model in one pipeline, so cross-validation refits preprocessing within each fold — preventing leakage (see Data Quality, Module 5).
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
print(cross_val_score(pipe, X_train, y_train, cv=5, scoring="roc_auc").mean().round(3))Key takeaways
- A large train–test gap signals overfitting.
- Random forests average many trees; boosting builds trees that correct errors.
- Cross-validation gives more reliable comparisons than one split.
- Tune hyperparameters with cross-validation; test once at the end.
- Pipelines refit preprocessing within each fold.