Module 3 of 6 · 35 minutes

Classification and evaluation

Yes or no? Fraud or not? Classification drives many business decisions — and choosing how to evaluate it is as important as choosing the model.

Learning objectives

Explain logistic regression and predicted probabilities
Train and read a decision tree
Interpret a confusion matrix, precision, recall, F1 and AUC
Choose a threshold based on business costs
Handle imbalanced classes

3.1 Logistic regression

Despite its name, logistic regression is a classification model. It combines features in a weighted sum, then squashes the result into a probability between 0 and 1. By default, probabilities above 0.5 become "yes".

proba = model.predict_proba(X_test)[:, 1]      # probability of churn for each customer
print(proba[:5].round(2))

3.2 Decision trees

A decision tree learns a flowchart of yes/no questions ("tenure under 8 months?", "more than 2 tickets?"). Trees are intuitive and capture non-linear patterns and interactions, but a single deep tree overfits easily (Module 4).

from sklearn.tree import DecisionTreeClassifier, export_text

tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
print(export_text(tree, feature_names=features))
print("Tree accuracy:", round(tree.score(X_test, y_test), 3))

A depth-2 decision tree: a learned flowchart of yes/no questions. (Illustrative splits — your tree's thresholds will differ.)

3.3 Beyond accuracy: the confusion matrix

from sklearn.metrics import confusion_matrix, classification_report

pred = model.predict(X_test)
print(confusion_matrix(y_test, pred))          # rows = actual, columns = predicted
print(classification_report(y_test, pred, digits=3))

Metric	Question
Precision	Of customers we flagged as churners, how many actually churned?
Recall	Of customers who churned, how many did we flag?
F1	A single score balancing precision and recall
ROC-AUC	How well the model ranks positives above negatives, across all thresholds (0.5 = random, 1 = perfect)

The confusion matrix: rows are what actually happened, columns are what the model predicted.

3.4 Choosing the threshold is a business decision

0.5 is just a default. If a retention call is cheap and losing a customer is expensive, flag more people (lower threshold, higher recall). If the team can only make 50 calls a week, flag only the most likely (higher threshold, higher precision).

from sklearn.metrics import precision_score, recall_score, roc_auc_score

print("ROC-AUC:", round(roc_auc_score(y_test, proba), 3))
for t in [0.3, 0.5, 0.7]:
    p = (proba >= t).astype(int)
    print(f"threshold {t}: precision {precision_score(y_test, p):.2f}  "
          f"recall {recall_score(y_test, p):.2f}  flagged {p.sum()}")

Each dot is a customer placed by predicted churn probability. Everyone right of the threshold gets flagged — moving the line trades recall against precision.

3.5 Imbalanced classes

When the positive class is rare (fraud, rare diseases), accuracy is misleading. Use precision, recall and AUC; stratify splits; consider class_weight="balanced"; and evaluate at the threshold you'll actually use.

Exercise: cost-based threshold

A missed churner costs £200; a retention call costs £10. Using proba, find the threshold between 0.05 and 0.95 that minimises total cost on the test set.

Show a solution

best = None
for t in np.arange(0.05, 0.96, 0.05):
    p = (proba >= t).astype(int)
    missed = ((p == 0) & (y_test == 1)).sum()
    calls = p.sum()
    cost = missed * 200 + calls * 10
    if best is None or cost < best[1]:
        best = (round(t, 2), cost)
print("Best threshold and cost:", best)

Key takeaways

Logistic regression outputs probabilities; 0.5 is just a default threshold.
Decision trees are readable flowcharts that can overfit.
Precision: of flagged, how many right? Recall: of real, how many caught?
Thresholds are business decisions — use costs and capacity.
With rare classes, ignore raw accuracy.