🤖 Section 5 · Machine Learning 🟡 Intermediate MODULE 27

Model Evaluation & Cross-Validation

⏱️ 50 min

📖 Beyond Accuracy

🧩 3 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 557%

🎯 "Accuracy" isn't always the whole story. Every classifier in Lessons 25 and 26 was checked with .score(), which reports accuracy — but accuracy can be dangerously misleading whenever one class is much rarer than the other. This lesson introduces precision, recall, and F1-score, the confusion matrix that all three come from, and k-fold cross-validation — a more reliable way to evaluate a model than trusting a single train/test split.

Section 1

Why Accuracy Alone Can Be Misleading

Accuracy is simply the fraction of predictions that were correct. That sounds like a fine summary — until the classes are imbalanced, meaning one class is far more common than the other.

⚠️

The classic imbalanced-class trap

Imagine a fraud-detection dataset where 95% of transactions are legitimate and only 5% are fraudulent. A "model" that does absolutely nothing useful — it just predicts "not fraud" for every single transaction, always — would score 95% accuracy. That number sounds impressive, but the model is completely useless: it catches zero fraud cases, which is the entire point of building it.

This is exactly why relying on accuracy alone is risky for imbalanced problems (fraud detection, rare disease diagnosis, spam filtering with mostly legitimate mail, and many more). Precision, recall, and F1-score exist specifically to surface what accuracy hides.

Section 2

The Confusion Matrix

Every classification metric in this lesson is built from four counts, laid out in a confusion matrix: how many predictions were correct, and how many were wrong, broken down by which way they were wrong.

	Predicted: Negative	Predicted: Positive
Actual: Negative	True Negative (TN)	False Positive (FP)
Actual: Positive	False Negative (FN)	True Positive (TP)

✅

True Positive (TP)

Model predicted positive, and it actually was positive. A correct "catch."

✅

True Negative (TN)

Model predicted negative, and it actually was negative. Correctly ignored.

❌

False Positive (FP)

Model predicted positive, but it was actually negative. A "false alarm" — Type I error.

❌

False Negative (FN)

Model predicted negative, but it was actually positive. A "miss" — Type II error, often the costlier mistake in cases like disease screening.

confusion_matrix_demo.py

PYTHON

from sklearn.metrics import confusion_matrix
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

cm = confusion_matrix(y_test, y_pred)
print(cm)
# Rows = actual class, columns = predicted class, in ascending label order
# e.g. [[TN, FP],
#       [FN, TP]]  for a 0/1 binary target

📝

Reading confusion_matrix()'s output

By default, confusion_matrix(y_true, y_pred) orders rows and columns by ascending label value — for a binary 0/1 target, row 0/column 0 is class 0, row 1/column 1 is class 1. Rows represent the true class; columns represent what the model predicted. Pass labels=[...] explicitly if you want to control or verify the ordering.

Section 3

Precision, Recall, and F1-Score

All three metrics answer a different, more specific question than plain accuracy — and all three are built directly from the confusion matrix counts above.

Precision — "Of everything I flagged as positive, how much was actually positive?"

Precision = TP / (TP + FP)

High precision means few false alarms. Matters most when a false positive is costly — e.g. flagging a legitimate transaction as fraud and blocking a real customer.

Recall — "Of everything that was actually positive, how much did I catch?"

Recall = TP / (TP + FN)

High recall means few missed positives. Matters most when a false negative is costly — e.g. missing an actual case of a disease during screening.

F1-Score — the harmonic mean of precision and recall

F1 = 2 × (Precision × Recall) / (Precision + Recall)

A single number balancing both — useful when you want one metric but neither precision nor recall alone tells the full story. The harmonic mean punishes a big imbalance between the two more than a simple average would.

⚠️

Precision and recall usually trade off against each other

A model that predicts "positive" for almost everything gets near-perfect recall (it catches nearly every real positive) but terrible precision (most of its positive predictions are false alarms). A model that only predicts "positive" when extremely confident gets high precision but low recall (it misses many real positives). Which one matters more is a business decision, not a purely statistical one — it depends on whether false positives or false negatives are more costly in the specific problem.

classification_report_demo.py

PYTHON

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred, target_names=data.target_names))

Illustrative classification_report() output

	precision	recall	f1-score	support
malignant	0.95	0.93	0.94	43
benign	0.96	0.97	0.97	71
accuracy			0.96	114
macro avg	0.96	0.95	0.95	114
weighted avg	0.96	0.96	0.96	114

✨

classification_report() computes everything at once, per class

classification_report() reports precision, recall, and F1 separately for every class (not just one overall number), plus support — how many true instances of each class were in the test set. That per-class breakdown is exactly what would have exposed the fraud-detection trap from Section 1: the majority class would look great, while the rare class's precision/recall would reveal the problem immediately. The numbers shown above are illustrative formatting, not a claimed result from any specific run.

Section 4

K-Fold Cross-Validation

Every metric so far was computed on a single train/test split — but that split was decided by one particular random_state. What if that one split happened to be unusually easy, or unusually hard? K-fold cross-validation fixes this by testing on several different splits and averaging the results.

Split the full dataset into K equal folds

Commonly K=5 or K=10 — e.g. with K=5, the data is divided into 5 roughly equal chunks.

Train on K−1 folds, test on the remaining one

With K=5: train on folds 1-4, test on fold 5. Record the score.

Rotate which fold is held out

Repeat Step 2 K times total, each time holding out a different fold as the test set.

Average the K scores

The final reported result is the mean (and often the standard deviation) across all K runs — a far more stable estimate than any single split.

cross_validation.py

PYTHON

from sklearn.model_selection import cross_val_score, KFold

model = RandomForestClassifier(n_estimators=100, random_state=42)

# shuffle=True randomizes row order before splitting into folds — important
# unless the rows are already in a random order
cv = KFold(n_splits=5, shuffle=True, random_state=42)

# cross_val_score handles the entire fit/predict/score loop across all 5 folds
scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy')

print("Scores per fold:", scores.round(3))
print(f"Mean: {scores.mean():.4f}  ±  Std: {scores.std():.4f}")

📝

cross_val_score() runs the WHOLE pipeline internally

Notice cross_val_score(model, X, y, ...) is passed the entire dataset, not a pre-split train/test pair — it handles the splitting, fitting, and scoring for all K folds internally, using a fresh, un-fitted copy of the model each time. There's no need to call .fit() yourself beforehand.

✨

The standard deviation is as informative as the mean

A small standard deviation across folds means the model performs consistently regardless of which rows end up in the test set — a good sign. A large standard deviation suggests the model's performance depends heavily on which specific rows it happens to see, which is itself a useful (and slightly concerning) finding that a single train/test split would never reveal.

🧩 Knowledge Check — Lesson 27

3 questions on evaluation metrics before you move on.

1. A dataset is 98% "not fraud" and 2% "fraud." A model that predicts "not fraud" for every single row scores 98% accuracy. What does this demonstrate?

2. A medical screening test has low recall for a disease. What does that mean in practice?

3. What is the main advantage of k-fold cross-validation over a single train/test split?

💪

Try It Yourself — Lesson 27

Evaluate properly · Intermediate Level

Reuse the breast cancer X, y, and the fitted model (RandomForestClassifier) from this lesson's code samples.

Task 1: Compute metrics by hand from a confusion matrix 🔢

Suppose a confusion matrix gives TP=40, FP=5, FN=8, TN=61. Calculate precision, recall, and F1-score by hand using the formulas from Section 3, then verify your numbers against sklearn.metrics.precision_score, recall_score, and f1_score.

Task 2: Try different values of K 🔁

Run cross_val_score with KFold(n_splits=k, shuffle=True, random_state=42) for k = 3, 5, and 10. Compare the mean and standard deviation across the three. Does a larger K change the mean much? Does it change the standard deviation?

Task 3: Compare cross-validation to a single split 🔍

Compute a single train/test split accuracy with random_state=1, then again with random_state=99. Compare both single-split numbers to the 5-fold cross-validation mean from Section 4. Which estimate would you trust more, and why?

💡 Show hints if you're stuck

Task 1: precision = 40/(40+5) = 0.889, recall = 40/(40+8) = 0.833, F1 = 2×(0.889×0.833)/(0.889+0.833) ≈ 0.860
Task 2: cross_val_score(model, X, y, cv=KFold(n_splits=3, shuffle=True, random_state=42)) — repeat for each K value.
Task 3: The cross-validation mean is generally the more trustworthy estimate — it isn't dependent on which particular rows landed in one lucky (or unlucky) test set.

Finished this lesson?

Mark it complete to track your progress.

Module 27 of 30 Section 5 — Machine Learning with Scikit-Learn

Model Evaluation & Cross-Validation

Why Accuracy Alone Can Be Misleading

The Confusion Matrix

Precision, Recall, and F1-Score

K-Fold Cross-Validation

Lesson 27 Complete!