📈 Section 4 · Statistics 🟡 Intermediate MODULE 19

Descriptive Statistics Deep Dive

⏱️ 45 min

📖 Mean, Spread & Outliers

🧩 3 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 420%

🎯 Welcome to Section 4. Every chart you built in Section 3 was a picture of numbers you could also describe precisely — where the data is centered, how spread out it is, and whether anything in it looks unusual. That precision is what descriptive statistics gives you: a small set of numbers that summarize a whole dataset. This lesson covers mean, median, mode, variance, standard deviation, quartiles, the IQR outlier method, and the one-line pandas summary, df.describe(), that computes almost all of them at once.

Section 1

Mean, Median & Mode — Measures of Central Tendency

All three answer the same question — "what's a typical value in this data?" — but they answer it differently, and picking the wrong one can make a dataset look like something it isn't.

➗

Mean

The sum of all values divided by how many there are. Sensitive to every value, including extreme ones.

🎯

Median

The middle value once the data is sorted. Barely affected by extreme values — robust to outliers.

🔁

Mode

The single most frequently occurring value. The only one of the three that also works on non-numeric (categorical) data.

⚖️

When they agree

In a roughly symmetric, bell-shaped distribution, mean ≈ median ≈ mode. It's when a distribution is skewed that they pull apart.

central_tendency.py

PYTHON

import numpy as np
import pandas as pd
from scipy import stats

# Monthly take-home pay for 9 employees at a small company, in PKR thousands
salaries = np.array([45, 48, 50, 52, 49, 51, 47, 53, 320])

print("Mean:  ", np.mean(salaries))
print("Median:", np.median(salaries))
print("Mode:  ", stats.mode(salaries, keepdims=True).mode[0])

# pandas Series has the same three as methods
s = pd.Series(salaries)
print(s.mean(), s.median(), s.mode()[0])

Output

Mean	79.4
Median	50.0
Mode	45 (each value here appears once — ties go to the smallest)

⚠️

When the mean lies to you: skewed data

Eight of nine employees earn between 45k and 53k PKR — the founder's 320k salary is an extreme value, an outlier. It drags the mean up to 79.4k, a number that describes no actual employee well. The median, 50k, is barely moved by that one value and reflects "typical pay" far more honestly. This is exactly the shape real income, house-price, and response-time data tends to take: a long right tail of a few very large values, called right-skewed (or positively skewed) data. As a rule of thumb, prefer the median over the mean whenever a distribution is skewed or has outliers.

📝

stats.mode() needs keepdims=True in modern SciPy

scipy.stats.mode() returns an object with .mode and .count arrays. Passing keepdims=True keeps its output shape consistent across SciPy versions — without it, older and newer SciPy releases return slightly different shapes for a 1D input.

Section 2

Variance & Standard Deviation — Measuring Spread

Central tendency alone can hide a lot. Two datasets can have the exact same mean while one is tightly clustered around it and the other is scattered wildly. Variance and standard deviation quantify that spread.

Population Variance

σ² = Σ(xᵢ − μ)² / N

Average of the squared distance from each value xᵢ to the population mean μ, over all N values.

Sample Variance (the one you'll use almost everywhere)

s² = Σ(xᵢ − x̄)² / (n − 1)

Same idea, but dividing by (n − 1) instead of n — Bessel's correction — because a sample tends to underestimate the true population spread otherwise.

Standard Deviation

σ = √(σ²) s = √(s²)

The square root of variance — back in the original units of the data, not squared units, which makes it far easier to interpret.

Why square the differences at all? Distances above and below the mean are a mix of positive and negative numbers that would otherwise cancel out to (or near) zero. Squaring makes every term positive before averaging, so spread in either direction adds up instead of canceling.

variance_stddev.py

PYTHON

scores_a = np.array([70, 72, 71, 69, 73])   # tightly clustered
scores_b = np.array([50, 90, 60, 85, 70])   # same mean, wide spread

print("Mean A:", np.mean(scores_a), " Mean B:", np.mean(scores_b))

# NumPy's np.var()/np.std() default to POPULATION (ddof=0)
print("Population variance A:", np.var(scores_a))
print("Population variance B:", np.var(scores_b))

# Pass ddof=1 to get the SAMPLE variance (divides by n-1 instead of n)
print("Sample variance A:", np.var(scores_a, ddof=1))
print("Sample std dev A: ", np.std(scores_a, ddof=1))

# pandas .var()/.std() default to SAMPLE (ddof=1) — the opposite default!
s_a = pd.Series(scores_a)
print("pandas sample variance A:", s_a.var())

⚠️

NumPy and pandas disagree on the default — this trips up everyone once

np.var() and np.std() default to ddof=0 (population, divide by N). pandas.Series.var() and .std() default to ddof=1 (sample, divide by n − 1). Same data, two libraries, two different default answers — until you pass ddof=1 to NumPy explicitly. In practice, almost all real datasets are samples, not full populations, so ddof=1 is usually the statistically correct choice regardless of which library you're in.

Section 3

Quartiles & the Interquartile Range (IQR)

Variance describes spread with one number derived from every value. Quartiles describe spread by cutting the sorted data into four equal-sized chunks, giving a more visual, order-based picture — the same idea behind the box plots from Lesson 15.

First quartile (25th percentile)

25% of values fall below this point.

Second quartile (50th percentile)

This is exactly the median — 50% of values fall below it.

Third quartile (75th percentile)

75% of values fall below this point.

Interquartile Range

IQR = Q3 − Q1

The range covered by the middle 50% of the data — a spread measure that, unlike variance, is barely affected by extreme outliers.

quartiles_iqr.py

PYTHON

data = np.array([12, 15, 14, 10, 18, 21, 13, 17, 16, 19])

q1 = np.percentile(data, 25)
q2 = np.percentile(data, 50)   # == np.median(data)
q3 = np.percentile(data, 75)
iqr = q3 - q1

print(f"Q1={q1}, Q2={q2}, Q3={q3}, IQR={iqr}")

# pandas: .quantile() uses a 0-1 scale instead of 0-100
s = pd.Series(data)
print(s.quantile([0.25, 0.5, 0.75]))

✨

np.percentile() uses 0–100, pandas .quantile() uses 0–1

They compute the same thing, just on different scales: np.percentile(data, 75) and pd.Series(data).quantile(0.75) both return the third quartile — mixing up the scale (asking for percentile 0.75 or quantile 75) is a common off-by-100 mistake.

Section 4

Detecting Outliers with the IQR Method

The IQR isn't just a spread measure — it's also the basis for the most common rule-of-thumb way to flag outliers, the same rule Matplotlib and Seaborn use to draw the individual dots beyond a box plot's whiskers.

IQR Outlier Bounds

lower bound = Q1 − 1.5 × IQR upper bound = Q3 + 1.5 × IQR

Any value outside [lower bound, upper bound] is flagged as an outlier. The 1.5 multiplier is a widely used convention, not a law of nature.

iqr_outliers.py

PYTHON

salaries = pd.Series([45, 48, 50, 52, 49, 51, 47, 53, 320])

q1 = salaries.quantile(0.25)
q3 = salaries.quantile(0.75)
iqr = q3 - q1

lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
print(f"Bounds: [{lower}, {upper}]")

outliers = salaries[(salaries < lower) | (salaries > upper)]
print("Outliers:\n", outliers)

# A clean version, with outliers removed
clean = salaries[(salaries >= lower) & (salaries <= upper)]
print("Clean mean:", clean.mean(), " vs original mean:", salaries.mean())

Output

Q1 / Q3 / IQR	48.0 / 52.0 / 4.0
Bounds	[42.0, 58.0]
Outliers	index 8 → 320
Clean mean vs original	50.6 vs 79.4

📝

The parentheses around each condition aren't optional

In (salaries < lower) | (salaries > upper), the parentheses are required because Python's operator precedence would otherwise evaluate | before </> — exactly the same boolean-masking rule from the NumPy lessons in Section 1, where &/| replace and/or on arrays and Series.

Section 5

df.describe() — All of This in One Line

pandas doesn't make you compute mean, std, and quartiles one at a time — df.describe() runs the whole summary in a single call, and it's usually the very first thing worth running on a new dataset, right alongside df.head() and df.info() from Section 2.

describe_demo.py

PYTHON

df = pd.DataFrame({
    "salary_k": [45, 48, 50, 52, 49, 51, 47, 53, 320],
    "years_exp": [2, 3, 4, 5, 3, 4, 2, 6, 12],
})

print(df.describe())

df.describe()

	salary_k	years_exp
count	9.0	9.0
mean	79.44	4.56
std	90.60	3.05
min	45.0	2.0
25%	47.0	3.0
50%	49.0	4.0
75%	52.0	5.0
max	320.0	12.0

✨

Reading a describe() table like a diagnosis

Compare mean (79.44) to 50% — the median (49.0) — for salary_k: they're far apart, and std (90.60) is enormous relative to the 25%–75% range. That combination, all by itself, is a strong sign of a right-skewed distribution with an outlier — exactly what Section 1's manual mean/median comparison found, now visible from one table without writing any of that code yourself.

📝

describe() skips non-numeric columns by default

Text or categorical columns are excluded automatically unless you pass df.describe(include="all"), which adds count, unique, top, and freq rows for those columns instead of mean/std/quartiles, which don't make sense for text.

🧩 Knowledge Check — Lesson 19

3 questions on descriptive statistics before you move on.

1. A dataset of house prices has a mean of $850,000 but a median of $410,000. What does this most likely indicate?

2. Why does sample variance divide by (n − 1) instead of n?

3. Using the standard IQR outlier rule, a value counts as an outlier if it falls:

💪

Try It Yourself — Lesson 19

Practice on your own data · Intermediate Level

Use this exam-scores sample as your starting point: scores = pd.Series([61, 74, 68, 72, 70, 65, 69, 200, 71, 73]) — note the 200, an obviously invalid data-entry error mixed in with real scores out of 100.

Task 1: Compare mean vs. median 📊

Compute scores.mean() and scores.median(). By how much does the invalid 200 entry distort the mean compared to the median?

Task 2: Flag it with the IQR method 🚩

Compute Q1, Q3, and IQR with .quantile(), then build the lower/upper bounds and use boolean masking to pull out every value scores flags as an outlier. Does it correctly catch the 200?

Task 3: Clean it and re-describe 🧹

Build a clean_scores Series with the outlier removed, then run .describe() on both the original and cleaned Series side by side. Which summary statistics change the most once the bad entry is gone?

💡 Show hints if you're stuck

Task 1: scores.mean() vs scores.median() — subtract one from the other for the exact distortion.
Task 2: q1, q3 = scores.quantile(0.25), scores.quantile(0.75), then scores[(scores < q1 - 1.5*(q3-q1)) | (scores > q3 + 1.5*(q3-q1))]
Task 3: clean_scores = scores[(scores >= lower) & (scores <= upper)], then pd.concat([scores.describe(), clean_scores.describe()], axis=1) to view both side by side.

Finished this lesson?

Mark it complete to track your progress.

Module 19 of 23 Section 4 — Statistics & Probability for Data Science

Descriptive Statistics Deep Dive

Mean, Median & Mode — Measures of Central Tendency

Variance & Standard Deviation — Measuring Spread

Quartiles & the Interquartile Range (IQR)

Detecting Outliers with the IQR Method

df.describe() — All of This in One Line

Lesson 19 Complete!