Text classification

Text classification routes, tags and monitors messages at scale. A TF-IDF pipeline is the fast, cheap baseline every text project should start with.

2.1 TF-IDF: weighting words that matter

Raw counts over-reward words that appear everywhere. TF-IDF (term frequency × inverse document frequency) raises the weight of words frequent in this message but rare across the collection, and lowers words that appear in almost every message.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True)
X_tfidf = tfidf.fit_transform(messages["text"])
print(X_tfidf.shape)

2.2 A text classifier in a pipeline

Put the vectoriser and model in one pipeline so the vocabulary is learned only from training data — the same leakage protection you met in Data Quality, Module 5.

Python · TF-IDF + logistic regressionfrom sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(
    messages["text"], messages["category"], test_size=0.25, random_state=0, stratify=messages["category"])

clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True),
                    LogisticRegression(max_iter=1000))
clf.fit(X_train, y_train)
print(classification_report(y_test, clf.predict(X_test), digits=3))

Expect high but not perfect scores: the 8% of noisy labels cap how well any model can appear to do. Look at the errors before assuming the model is wrong — sometimes the label is.

2.3 What did the model learn?

vec = clf.named_steps["tfidfvectorizer"]
lr = clf.named_steps["logisticregression"]
words = vec.get_feature_names_out()
for i, cat in enumerate(lr.classes_):
    top = np.argsort(lr.coef_[i])[-6:][::-1]
    print(f"{cat:9}", ", ".join(words[top]))

Inspecting the strongest words is a fast sanity check. If a "billing" model relies on a customer's name or a date format, it has learned a shortcut that won't generalise.

2.4 Reading the errors

test = pd.DataFrame({"text": X_test, "actual": y_test, "predicted": clf.predict(X_test)})
errors = test[test["actual"] != test["predicted"]]
print(len(errors), "errors")
print(errors.head(8).to_string())

Sort errors into buckets: wrong label, genuinely ambiguous message, or model mistake. Only the last bucket is fixed by a better model; the first is fixed by better labels (see Data Quality, Module 4).

2.5 Sentiment analysis

Sentiment is just classification with labels like positive, negative and neutral. The same pipeline works if you have labelled examples. Without them, an LLM can label a sample for you to check (Module 5) — or you can use an off-the-shelf sentiment model. Watch for sarcasm, negation and domain words ("sick" can be praise).

new = ["My kettle stopped working", "Charged twice, please refund", "Courier lost my parcel",
       "The lamp arrived late and I was billed twice"]
for text, probs in zip(new, clf.predict_proba(new)):
    best = probs.argmax()
    print(f"{clf.classes_[best]:9} {probs[best]:.2f}  {text}")
# The mixed message gets a lower top probability: a good candidate for human review.

Key takeaways

• TF-IDF up-weights distinctive words and down-weights ubiquitous ones.
• Fit the vectoriser inside a pipeline to avoid leakage.
• Check the strongest words for shortcuts.
• Some 'errors' are wrong labels — read them.
• Sentiment is classification with care for negation and sarcasm.