🎯 Section 3 · Model Evaluation Mastery 🟡 Intermediate MODULE 14

Dealing with Imbalanced Datasets

⏱️ 24 min read

📖 Class Weighting, SMOTE & Undersampling

🧩 4 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 325%

🎯 What you'll learn: Lesson 13 warned that accuracy gets "especially unreliable with imbalanced classes" — this lesson is that warning made concrete. You'll see exactly HOW a naive model exploits an imbalanced dataset to score deceptively high accuracy, then fix it three different ways: class_weight="balanced" built into scikit-learn estimators, oversampling the minority class with SMOTE, undersampling the majority class with RandomUnderSampler, and finally which metrics (F1, precision-recall AUC) actually tell you the truth on skewed data.

Section 1

Why Imbalanced Classes Break Naive Accuracy

A dataset is "imbalanced" when one class vastly outnumbers another. The classic example: fraud detection, where the overwhelming majority of transactions are legitimate and only a small fraction are fraudulent. The illustrative numbers below are a common teaching setup, not a claim about any real payment processor's actual fraud rate.

An illustrative imbalanced dataset — fraud_transactions.csv (10,000 rows)

class	row count	% of dataset
Legitimate (0)	9,700	97.0%
Fraud (1)	300	3.0%

Now imagine the laziest possible "model" — one that ignores every feature and always predicts 0 (legitimate):

the_accuracy_trap.py

PYTHON

import numpy as np
from sklearn.metrics import accuracy_score, recall_score

# A "dumb" prediction array: always predicts the majority class (0)
dumb_preds = np.zeros_like(y_test)

print("Accuracy:", accuracy_score(y_test, dumb_preds))
print("Recall on fraud class:", recall_score(y_test, dumb_preds))

# Accuracy: 0.970   -> looks great!
# Recall on fraud class: 0.0   -> catches ZERO fraud, ever

⚠️

97% accuracy, 0% usefulness

This "model" scores higher accuracy than most real, carefully-trained classifiers would — while being completely useless for the actual task of catching fraud. This is precisely why Lesson 13 introduced precision, recall, and F1: on imbalanced data, accuracy can hide a model that has learned nothing at all about the minority class.

Section 2

Fix 1 — Class Weighting

Many scikit-learn classifiers accept a class_weight parameter. Setting it to "balanced" tells the model to penalize mistakes on the minority class more heavily during training — automatically, based on how imbalanced the classes actually are.

class_weighting.py

PYTHON

from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier

# class_weight="balanced" works on LogisticRegression, SVC, RandomForestClassifier,
# DecisionTreeClassifier, and several other sklearn estimators
log_reg = LogisticRegression(class_weight="balanced", random_state=42)
svc = SVC(class_weight="balanced", probability=True, random_state=42)
rf = RandomForestClassifier(class_weight="balanced", random_state=42)

log_reg.fit(X_train, y_train)
print("Recall on fraud class:", recall_score(y_test, log_reg.predict(X_test)))
# Recall on fraud class: 0.79   -> a real, useful improvement over 0.0

📝

What "balanced" actually computes

Internally, class_weight="balanced" sets each class's weight to n_samples / (n_classes * count_of_that_class) — so the rare class (fraud, at 3%) gets a much larger weight than the common class (legitimate, at 97%), making mistakes on fraud "count more" during training. No resampling of rows happens at all — the data itself is untouched; only how much each row's error matters changes.

✨

The cheapest fix to try first

Because class_weight="balanced" is a single keyword argument with no extra preprocessing, no new library, and no risk of accidentally leaking synthetic data into a test set, it's usually the first thing worth trying on an imbalanced problem before reaching for SMOTE or undersampling below.

Section 3

Fix 2 — Oversampling the Minority Class with SMOTE

SMOTE (Synthetic Minority Over-sampling Technique), from the separate imbalanced-learn package (imblearn), generates NEW synthetic examples of the minority class rather than simply duplicating existing rows — it interpolates between real minority-class points and their nearest minority-class neighbors.

smote_oversampling.py

PYTHON

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Split FIRST — SMOTE must only ever see training data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

print("Before SMOTE:", y_train.value_counts().to_dict())
# Before SMOTE: {0: 7760, 1: 240}

smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

print("After SMOTE:", y_train_resampled.value_counts().to_dict())
# After SMOTE: {0: 7760, 1: 7760}   -> minority class synthetically balanced to match

model = LogisticRegression(random_state=42)
model.fit(X_train_resampled, y_train_resampled)

# Evaluate on the REAL, untouched, still-imbalanced test set
print("Recall on fraud class:", recall_score(y_test, model.predict(X_test)))

⚠️

Never apply SMOTE before splitting, and never resample the test set

Two rules, both non-negotiable: (1) train_test_split must run BEFORE fit_resample, so no synthetic points end up leaking real minority-class information into the test set. (2) The TEST set is always evaluated in its original, real, imbalanced form — resampling it would mean testing on fabricated data instead of measuring real-world performance. Only X_train/y_train ever get passed to fit_resample.

📝

stratify=y keeps the split's class proportions honest

stratify=y in train_test_split ensures both the train and test sets preserve the SAME class imbalance ratio as the full dataset — important on already-rare classes, where a random split could otherwise leave very few (or zero) minority examples in the test set by chance.

Section 4

Fix 3 — Undersampling the Majority Class

The opposite approach: instead of manufacturing more minority-class rows, randomly DROP majority-class rows until the classes are balanced. Also from imbalanced-learn:

random_undersampling.py

PYTHON

from imblearn.under_sampling import RandomUnderSampler

# Same rule as SMOTE: only ever call this on training data, after splitting
rus = RandomUnderSampler(random_state=42)
X_train_resampled, y_train_resampled = rus.fit_resample(X_train, y_train)

print("Before:", y_train.value_counts().to_dict())
print("After: ", y_train_resampled.value_counts().to_dict())
# Before: {0: 7760, 1: 240}
# After:  {0: 240, 1: 240}   -> majority class trimmed DOWN to match the minority

🧬

SMOTE (oversampling)

Generates synthetic minority examples. Keeps every original row. Works well when the dataset isn't huge and you can't afford to throw data away.

✂️

RandomUnderSampler (undersampling)

Drops random majority-class rows. Simpler and faster, but throws away real data — risky if the majority class already had limited examples of its own diversity.

⚠️

Undersampling can discard useful signal

If the majority class only has, say, 300 training rows to begin with, undersampling it down to match a 240-row minority class leaves very little data overall to learn from. SMOTE tends to be the safer default on smaller datasets for exactly this reason — though on very LARGE datasets, undersampling the majority class can be an efficient, practical choice that also speeds up training.

Section 5

Choosing the Right Metric for Imbalanced Data

Resampling and class weighting help a model LEARN better on imbalanced data — but you also need a metric that can actually SEE whether it worked. Section 1 already showed accuracy can't. From Lesson 13's toolbox, three are far more trustworthy here.

🎯

F1-Score

Balances precision and recall in one number — stays low if the model ignores the minority class, unlike accuracy.

📈

Precision-Recall AUC

Like ROC-AUC but plots precision vs recall across thresholds — more informative than ROC-AUC specifically when the positive class is rare.

📋

classification_report

Shows per-class precision/recall/F1 side by side, making a model that's ignoring the minority class immediately visible.

imbalanced_metrics.py

PYTHON

from sklearn.metrics import f1_score, average_precision_score, classification_report

y_probs = model.predict_proba(X_test)[:, 1]
y_pred = model.predict(X_test)

print("F1-score (fraud class):", f1_score(y_test, y_pred))
print("Precision-Recall AUC:", average_precision_score(y_test, y_probs))
print(classification_report(y_test, y_pred, target_names=["Legitimate", "Fraud"]))

Illustrative comparison — same toy dataset, three approaches

Approach	Accuracy	Recall (fraud)	F1 (fraud)
Always predict "legitimate"	0.970	0.000	0.000
class_weight="balanced"	0.941	0.790	0.583
SMOTE + LogisticRegression	0.936	0.810	0.571

✨

Notice accuracy actually goes DOWN when the model improves

In this illustrative comparison, both real fixes score LOWER accuracy than the useless "always predict legitimate" baseline — while being dramatically more useful, catching roughly 79-81% of actual fraud instead of none. This is the single clearest demonstration in the whole lesson of why accuracy alone cannot be trusted to judge a classifier on imbalanced data.

Section 6

Lesson Summary

✅A model that always predicts the majority class can score deceptively high accuracy while being completely useless.

✅class_weight="balanced" is the cheapest fix — no resampling, just reweighted training errors.

✅SMOTE generates synthetic minority examples; RandomUnderSampler drops majority examples — both fit only on training data, AFTER splitting.

✅Always evaluate on the real, untouched, still-imbalanced test set — never resample it.

✅Judge imbalanced classifiers with F1, precision-recall AUC, or classification_report — never plain accuracy alone.

🧩 Knowledge Check — Lesson 14

4 questions on imbalanced classes, class weighting, SMOTE, and undersampling.

1. A dataset is 97% legitimate / 3% fraud. A model that always predicts "legitimate" scores 97% accuracy. What does its recall on the fraud class look like?

2. What does class_weight="balanced" actually do to the training data?

3. Where should train_test_split happen relative to SMOTE's fit_resample?

4. Why is F1-score generally more trustworthy than accuracy on imbalanced data?

💪

Try It Yourself — Lesson 14

Fix an imbalanced classifier three different ways · Intermediate Level

Take (or construct) any imbalanced binary classification dataset — even an artificially imbalanced version of a dataset you've already used in this course by dropping most rows of one class.

Task 1: Baseline vs. class_weight="balanced" ⚖️

Train a LogisticRegression with default settings, then again with class_weight="balanced". Compare recall_score and f1_score for the minority class between the two. Confirm the balanced version catches more of the minority class.

Task 2: SMOTE it up 🧬

Split your data with stratify=y, apply SMOTE().fit_resample() to the TRAINING data only, train a fresh model on the resampled data, and evaluate it on the original, untouched test set. Print class counts before and after SMOTE to confirm the training data became balanced.

Task 3: Three-way comparison table 📊

Build a small pandas DataFrame comparing the untouched baseline, the class_weight="balanced" version, and the SMOTE version, with columns for accuracy, precision, recall, and F1 (all on the minority class). Write one sentence: which approach would you actually deploy, and why?

💡 Show hints if you're stuck

Task 1: If recall barely changes, double check class_weight="balanced" was actually passed and the model was refit — it has no effect unless retrained with it.
Task 2: y_train.value_counts() (if y_train is a pandas Series) should show two roughly equal counts after fit_resample, and two very unequal counts before.
Task 3: There's often no single "correct" answer — the right choice depends on which metric (recall vs. precision) matters more for your specific problem, exactly as Lesson 13 Section 6 discussed.

Finished this lesson?

Mark it complete to track your progress.

Module 14 of 24 Section 3 — Model Evaluation Mastery

Dealing with Imbalanced Datasets

Why Imbalanced Classes Break Naive Accuracy

Fix 1 — Class Weighting

Fix 2 — Oversampling the Minority Class with SMOTE

Fix 3 — Undersampling the Majority Class

Choosing the Right Metric for Imbalanced Data

Lesson Summary

Lesson 14 Complete!