Machine Learning Fundamentals / Section 1 — What is Machine Learning? / Lesson 5
🧠 Section 1 · Foundations 🟡 Intermediate MODULE 05

Overfitting vs Underfitting

⏱️ 24 min read
📖 Model Behavior
🧩 4 Quiz Questions
🏗️ 1 Challenge · 3 tasks
Your progress in Section 1100%
🎯 What you'll learn: This is the last conceptual lesson before Section 2's algorithms, and arguably the most important one in the whole course. You'll build real intuition for the bias-variance tradeoff, learn to recognize overfitting and underfitting from train-vs-test scores, see regularization (Ridge, Lasso) as a direct countermeasure, and understand what a learning curve is telling you.
Section 1

The Bias-Variance Tradeoff, Intuitively

Every model's error comes from two competing sources. Bias is error from a model being too simple to capture the real pattern — it makes systematic mistakes no matter how much data you give it. Variance is error from a model being too sensitive to the specific training data it happened to see — it would produce a wildly different model if trained on a slightly different sample. The tradeoff: reducing one tends to increase the other, and the best models sit at a sweet spot in between.

🎓
The exam-studying analogy
Picture two students preparing with last year's practice exam. One memorizes the exact answers to every practice question, word for word — high "bias-free" performance on that exact test, but they collapse the moment a real question is phrased even slightly differently. That's high variance: their knowledge doesn't generalize, it's overfit to one specific practice set. The other student skims the material without really engaging — they do poorly on the practice test AND the real exam, because they never learned the underlying concepts in the first place. That's high bias: their understanding is too crude to capture the real pattern, no matter which exam they take. The student who does best studied enough to understand the concepts generally, without memorizing one specific set of questions — low bias AND low variance.
The Balance You're Looking For
High Bias (underfit)  ⟷  Sweet Spot  ⟷  High Variance (overfit)
Model complexity is usually the dial that moves you along this line — too simple underfits, too complex overfits.
Section 2

Spotting Overfitting and Underfitting

In practice, you diagnose which regime a model is in by comparing its score on the training set to its score on the test set.

📚
Overfitting
Training score is HIGH, test score is noticeably LOWER. The model memorized noise and quirks specific to the training rows instead of the general pattern.
😴
Underfitting
Both training score AND test score are LOW and close together. The model is too simple to capture the real relationship in the data at all.
🎯
Good fit
Training and test scores are both reasonably HIGH and close together. The model generalizes — it learned the pattern, not the noise.
🌡️
Model complexity is the dial
A very shallow decision tree tends to underfit; an unrestricted, very deep one tends to overfit. Section 2 onward will show this pattern repeat across different algorithm families.
📈 Illustrative pattern — train vs. test score as model complexity increases
Picture the x-axis as "model complexity" (e.g. decision tree depth) increasing left to right, and the y-axis as score. The training score line climbs steadily and keeps climbing — a more complex model can always fit its own training data better. The test score line climbs at first (the model is learning real signal), peaks, then starts falling as complexity keeps increasing past that peak (the model starts fitting noise instead). The gap that opens up between the two lines after the peak IS overfitting, visually. The region on the far left, where BOTH lines are low, is underfitting.
Illustrative training vs. test scores across three regimes
Underfit — train
0.58
low
Underfit — test
0.55
low
Good fit — train
0.88
high
Good fit — test
0.85
high
Overfit — train
0.99
high
Overfit — test
0.62
low

Illustrative numbers, not measured results — the pattern to notice is the SIZE OF THE GAP between train and test, not any specific score.

⚠️
A perfect training score is a warning sign, not a triumph
If a model scores close to 100% on training data, that's often a red flag rather than good news — check the test score before celebrating. This is exactly why Lesson 4's train/test split and cross-validation exist: without a held-out evaluation, overfitting would be invisible until the model failed on real data.
Section 3

Regularization as a Countermeasure

Regularization discourages a model from fitting the training data too closely by adding a penalty for complexity into what the model optimizes. For linear models, the two standard forms are:

Ⓛ
L2 regularization (Ridge)
Penalizes the sum of squared coefficients, shrinking all coefficients toward zero (but rarely exactly to zero). Tends to handle correlated features gracefully.
Ⓛ
L1 regularization (Lasso)
Penalizes the sum of absolute coefficients, which can push some coefficients to EXACTLY zero — effectively performing feature selection by dropping the least useful features.
regularization.py
PYTHON 
from sklearn.linear_model import Ridge, Lasso

# alpha controls regularization strength — higher alpha = more penalty = simpler model
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)

lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)

print("Ridge test score:", ridge.score(X_test, y_test))
print("Lasso test score:", lasso.score(X_test, y_test))
print("Lasso coefficients (some may be exactly 0):", lasso.coef_)
✨
alpha is a hyperparameter, not something the model learns
Recall from Lesson 2's pipeline stages: alpha is tuned, not fit. Too small and it barely regularizes (still overfits); too large and the model shrinks toward predicting the average, ignoring real signal (underfits). Lesson 4's cross-validation is exactly the tool used to pick a good alpha — try several values and compare their cross-validated scores.

Regularization isn't unique to linear models — tree-based models have their own complexity controls (like maximum depth), and later lessons in this course will cover the equivalent knobs for each algorithm family as they're introduced.

Section 4

Learning Curves

A learning curve plots model performance against the AMOUNT of training data used, instead of against model complexity. scikit-learn's learning_curve function automates generating one: it trains the model repeatedly on growing subsets of the training data and records both training and cross-validation scores at each size.

learning_curve.py
PYTHON 
from sklearn.model_selection import learning_curve
import numpy as np

train_sizes, train_scores, val_scores = learning_curve(
    model, X, y,
    cv=5,
    train_sizes=np.linspace(0.1, 1.0, 5)   # 10%, 32.5%, 55%, 77.5%, 100% of training data
)

print("Training set sizes used:", train_sizes)
print("Mean training score at each size:", train_scores.mean(axis=1))
print("Mean validation score at each size:", val_scores.mean(axis=1))
📉
Underfitting on a learning curve
Both curves converge to a low score and stay close together, even with more data — more training rows won't fix a model that's fundamentally too simple.
📈
Overfitting on a learning curve
A large, persistent gap between a high training score and a lower validation score — though the gap often narrows as more training data is added, since it's harder to memorize a bigger dataset.
📝
What a learning curve tells you that a single train/test comparison doesn't
It answers a very practical question: "would collecting more data actually help?" If the validation curve is still rising and the gap to the training curve is narrowing as data increases, more data likely helps. If both curves have flattened out with a persistent gap, the fix is more likely a different model or feature engineering, not more rows.
Section 5

Lesson Summary — and Section 1 Wrap-Up

✅The bias-variance tradeoff: too simple underfits (high bias), too complex overfits (high variance).
✅Overfitting = high train score, much lower test score. Underfitting = both scores low and close together.
✅Regularization (Ridge = L2, Lasso = L1) penalizes model complexity to fight overfitting; alpha controls the strength.
✅Learning curves (learning_curve) show whether more training data would actually help.
✅Section 1 complete: you now have the full conceptual toolkit — ML types, the pipeline, preprocessing, splitting/cross-validation, and overfitting — that every algorithm in Section 2 onward builds on.
🧩 Knowledge Check — Lesson 5
4 questions on bias-variance, overfitting/underfitting, and regularization.
1. A model scores 0.97 on training data and 0.55 on test data. What does this most likely indicate?
2. A model scores 0.52 on training data and 0.50 on test data. What does this indicate?
3. Which regularization technique can shrink some coefficients to EXACTLY zero, effectively performing feature selection?
4. On a learning curve, both the training and validation scores are low and have flattened out together, even with more data. What does this suggest?
💪
Try It Yourself — Lesson 5
Diagnose fit and apply regularization · Intermediate Level

These tasks tie together everything from Section 1 before Section 2 begins.

Task 1: Diagnose three models 🔍

For each pair of (train score, test score), label it overfitting, underfitting, or good fit, and explain your reasoning in one sentence: (a) 0.91 / 0.89, (b) 0.60 / 0.58, (c) 0.98 / 0.71.
Task 2: Compare Ridge at two alpha values 🎛️

Using the code sample in Section 3, fit Ridge(alpha=0.01) and Ridge(alpha=100) on the same X_train/y_train. Compare their .score() on both the training and test sets. Which one looks more likely to be underfitting?
Task 3: Write your own analogy 🎓

Section 1 used the exam-studying analogy for bias-variance. Write 2–3 sentences describing overfitting and underfitting using a COMPLETELY different everyday analogy (cooking, sports, navigation, anything) — matching the same shape: one extreme that's too rigid/simple, one that's too specific/memorized, and a middle ground that generalizes.
💡 Show hints if you're stuck
  • Task 1: (a) good fit — both high and close. (b) underfitting — both low and close. (c) overfitting — high train, noticeably lower test.
  • Task 2: A very large alpha (100) pushes coefficients heavily toward zero, which tends to hurt EVEN the training score if it's too aggressive — a sign of underfitting from too much regularization.
  • Task 3: A good analogy example: a chef who only ever cooks one exact recipe from memory (overfit) vs. one who never learned any techniques and guesses randomly (underfit) vs. one who understands techniques well enough to adapt to new ingredients (good fit).
Finished this lesson?
Mark it complete to track your progress.
🎉

Section 1 Complete!

You now have the full conceptual foundation: what ML is, the standard pipeline, preprocessing, splitting/cross-validation, and overfitting vs underfitting. Next up: Section 2 starts with real algorithms, beginning with the theory and math behind linear regression.

Module 05 of 24 Section 1 — What is Machine Learning?