Text as data

Before any model can work with language, text has to become numbers. This module shows the simplest ways to do that — and the choices that matter.

1.1 What is natural language processing?

Natural language processing (NLP) is the set of techniques that let computers work with human language: classifying messages, finding information, measuring sentiment, searching by meaning, translating, summarising and generating text. Large language models are the newest and most powerful NLP tools — but classic techniques remain fast, cheap and surprisingly strong for many business tasks.

1.2 The practice dataset

You'll use 400 synthetic customer messages, each labelled delivery, product or billing. About 8% of the labels are deliberately wrong — real labels are never perfect.

Python · practice data: synthetic customer messages (run this first)import random
import re
import numpy as np
import pandas as pd

random.seed(7)
templates = {
    "delivery": ["My parcel arrived {d} days late and the box was crushed",
                 "Delivery took {d} days and the courier left it in the rain",
                 "Still waiting for my order after {d} days, no update",
                 "The tracking link never updated and delivery was {d} days late"],
    "product":  ["The {p} broke after a week",
                 "Love the {p}, great quality and well made",
                 "The {p} is much smaller than the photos suggest",
                 "Excellent {p}, works perfectly"],
    "billing":  ["I was charged twice for order {o}",
                 "Refund for order {o} still not received",
                 "The discount code did not apply at checkout",
                 "Invoice for order {o} shows the wrong amount"],
}
products = ["chopping board", "kettle", "desk lamp", "backpack", "water bottle"]
rows = []
for i in range(400):
    cat = random.choice(list(templates))
    msg = random.choice(templates[cat]).format(
        d=random.randint(2, 9), p=random.choice(products), o="GFY-%05d" % random.randint(10000, 99999))
    label = cat if random.random() > 0.08 else random.choice(list(templates))  # ~8% noisy labels
    rows.append({"text": msg, "category": label})
messages = pd.DataFrame(rows)
print(messages.sample(5, random_state=1))
print(messages["category"].value_counts())

1.3 From text to tokens

Computers can't do arithmetic on sentences, so the first step is to break text into tokens — usually words or pieces of words — and often to normalise them (lowercase, remove punctuation).

def simple_tokens(text):
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s-]", " ", text)   # keep letters, digits, spaces, hyphens
    return text.split()

print(simple_tokens("Refund for order GFY-12345 still NOT received!"))

Choices like these matter. Removing punctuation helps matching but can destroy meaning ("not" vs "not!" is harmless; "can't" becoming "can t" is not). Lowercasing merges "Apple" the company with "apple" the fruit. There's no universally right answer — check what your task needs.

1.4 Bag of words

The simplest way to turn text into numbers is a bag of words: count how often each word appears, ignoring order. Each message becomes a row; each word in the vocabulary becomes a column.

from sklearn.feature_extraction.text import CountVectorizer

docs = ["parcel arrived late", "refund not received", "parcel late again"]
cv = CountVectorizer()
counts = cv.fit_transform(docs)
print(pd.DataFrame(counts.toarray(), columns=cv.get_feature_names_out()))

Bag of words loses word order — "not good, bad" and "not bad, good" look identical — but for many classification tasks it works remarkably well.

1.5 Stop words and n-grams

Stop words are very common words ("the", "and", "for") that often add little meaning. N-grams keep short sequences of words together, so "not received" or "charged twice" become features in their own right — partly recovering word order.

cv2 = CountVectorizer(stop_words="english", ngram_range=(1, 2), min_df=3)
X_counts = cv2.fit_transform(messages["text"])
print(X_counts.shape)                                  # (messages, features)
top = np.asarray(X_counts.sum(axis=0)).ravel()
print(pd.Series(top, index=cv2.get_feature_names_out()).sort_values(ascending=False).head(10))

Be careful with stop-word lists: scikit-learn's English list includes "not" and "no", which flip meaning in sentiment tasks.

for cat, group in messages.groupby("category"):
    m = cv2.transform(group["text"])
    freq = pd.Series(np.asarray(m.sum(axis=0)).ravel(), index=cv2.get_feature_names_out())
    print(cat, list(freq.sort_values(ascending=False).head(5).index))

Key takeaways

• NLP covers classification, extraction, search, topics and generation.
• Tokenisation and normalisation choices change meaning — test them.
• Bag of words counts words and ignores order.
• N-grams recover some word order; stop-word lists can remove 'not'.