Module 2 of 6 · 30 minutes

Profiling and exploring data

Measure before you change. Profiling turns a vague sense that 'the data's a bit messy' into a precise list of problems.

Learning objectives

Profile structure, types, missing values and duplicates
Use value counts to find inconsistent categories
Find impossible values and statistical outliers
Check validity of formats such as dates
Write a data quality report

2.1 Profile before you change anything

Data profiling means systematically measuring a dataset: its size, types, missing values, distributions, unique values and relationships. It tells you what needs fixing — and gives you a baseline to compare against after cleaning.

2.2 A first look

Python · structural profileprint(customers.shape)            # (rows, columns)
customers.info()                  # types and non-null counts
print(customers.isna().sum())     # missing values per column
print(customers.isna().mean().round(2))   # share missing
print(customers.duplicated().sum(), "fully duplicated rows")
print(customers["customer_id"].duplicated().sum(), "repeated IDs")

2.3 Categorical columns

for col in ["country", "churned"]:
    print(customers[col].value_counts(dropna=False), "\n")

Value counts instantly expose inconsistent spellings and unexpected categories.

2.4 Numerical columns and outliers

print(customers[["age", "monthly_spend"]].describe())

# Rule-based checks: values outside what's possible
print(customers[(customers["age"] < 16) | (customers["age"] > 110)])

# Statistical check: the IQR rule flags unusually large or small values
q1, q3 = customers["monthly_spend"].quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(customers[(customers["monthly_spend"] < low) | (customers["monthly_spend"] > high)])

The IQR rule flags values far beyond the middle half of the data. Flagged values are leads to investigate — not automatic deletions.

Outlier ≠ error

A value can be unusual but real — a genuine big-spending customer. Use rules (an age of 230 is impossible) to find errors, and statistics to find values worth investigating. Ask someone who knows the business before deleting anything.

2.5 Validity checks

Python · which dates don't parse?parsed = pd.to_datetime(customers["signup_date"], format="%Y-%m-%d", errors="coerce")
bad_dates = customers.loc[parsed.isna() & customers["signup_date"].notna(), "signup_date"]
print(bad_dates)      # "04/02/2025" (different format) and "2025-13-01" (invalid month)

2.6 Visual profiling

import matplotlib.pyplot as plt

customers["monthly_spend"].plot(kind="hist", bins=20, title="Monthly spend")
plt.show()
customers.boxplot(column="age")
plt.show()

Histograms reveal skew and spikes (e.g. lots of zeros or a default like 999); box plots show outliers at a glance.

2.7 Write a data quality report

Summarise each issue: the column, the problem, how many rows, the likely cause, and the proposed fix. Share it with the data's owners — they often know causes you can't see (a system migration, a form default, a manual entry step).

Exercise: a reusable profile function

Write a function profile(df) that returns a DataFrame with one row per column showing its type, number missing, percentage missing and number of unique values.

Show a solution

def profile(df):
    return pd.DataFrame({
        "dtype": df.dtypes.astype(str),
        "missing": df.isna().sum(),
        "missing_pct": (df.isna().mean() * 100).round(1),
        "unique": df.nunique(dropna=True),
    })

print(profile(customers))

Key takeaways

info(), isna() and duplicated() give a structural profile.
value_counts exposes inconsistent categories.
Rules find errors; statistics find values to investigate.
Coercing dates reveals invalid and mixed formats.
Share a data quality report with data owners.