🎯 Section 3 · Model Evaluation Mastery 🔴 Capstone Project MODULE 16

Project — ML Model Comparison Report

⏱️ 85 min · hands-on

📖 Cross-Validated Model Benchmarking

🧩 4 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 3100%

🎯 The Project: This is the Section 3 capstone — every evaluation tool from Lessons 13–15 gets put to work on a real decision. You'll take the house_prices.csv dataset from Lesson 12, systematically cross-validate FOUR very different regression models from Section 2 — LinearRegression, SVR, KNeighborsRegressor, and GradientBoostingRegressor — using cross_validate, build a real comparison table with MAE/RMSE/R² (Lesson 15), run residual analysis on the winner, and write a short recommendation of which model you'd actually deploy and why.

Section 1

The Project Brief

Lesson 12 trained exactly two models — LinearRegression and GradientBoostingRegressor — on one train/test split, and picked a winner. That's a reasonable first pass, but Lesson 11 already warned that a single split is "just one estimate," and Section 2 covered several more algorithms that never even got a turn. This project fixes both gaps at once: more models, evaluated the honest way.

📋 The brief

Reusing the same illustrative house_prices.csv dataset from Lesson 12 (sqft, bedrooms, bathrooms, city, age_years → price), benchmark four regression algorithms from Section 2 with 5-fold cross-validation, rank them on MAE, RMSE, and R², inspect the winner's residual plot, and write a one-paragraph recommendation of which model to ship.

📈

LinearRegression

Lesson 6. Simple, fast, fully interpretable coefficients. Assumes a linear relationship.

🧭

SVR (Support Vector Regression)

Lesson 7's SVM, adapted for regression. Can capture non-linear patterns via its kernel, but sensitive to feature scale and hyperparameters.

📍

KNeighborsRegressor

Lesson 8's KNN, adapted for regression — predicts the average price of the k nearest houses in feature space. No training phase, but slower at prediction time.

🌲

GradientBoostingRegressor

Lesson 10. Sequential ensemble of shallow trees. Captures non-linear relationships, generally strong out of the box.

Load and preprocess

Reuse Lesson 12's cleaning, encoding, and feature/target split.

Define four candidate models

One estimator each, wrapped in its own preprocessing pipeline.

Cross-validate all four with cross_validate

cv=5, scoring for RMSE, MAE, and R² simultaneously (Lesson 11).

Build the comparison table

A small pandas DataFrame, sorted by RMSE, with mean AND standard deviation per model.

Residual-check the winner

Predicted-vs-actual and residuals-vs-predicted plots (Lesson 15) on the top model.

Write the recommendation

Which model to deploy, and why — weighing accuracy against training time and interpretability.

Section 2 · Step 1

Load and Preprocess — Reusing Lesson 12's Pipeline

No need to reinvent this part — Lesson 12 already worked out the cleaning and encoding steps for this exact dataset. The only change here: features are kept UNSCALED at this stage, because each model below will get its own scaler inside its own pipeline in Section 3, to keep cross-validation leakage-free per fold.

load_and_prep.py

PYTHON

import pandas as pd

df = pd.read_csv("house_prices.csv")

# Same cleaning as Lesson 12 Section 3
df["bedrooms"] = df["bedrooms"].fillna(df["bedrooms"].median())
assert df.isnull().sum().sum() == 0

# Same encoding as Lesson 12 Section 4
df_encoded = pd.get_dummies(df, columns=["city"], drop_first=True)

X = df_encoded.drop(columns=["price"])
y = df_encoded["price"]

print(f"X: {X.shape}, y: {y.shape}")
# X: (1500, 7), y: (1500,)

📝

No train_test_split here — cross_validate does its own splitting

Unlike Lesson 12's single-split comparison, this project passes the FULL X/y straight to cross_validate in Section 4. Internally it performs its own 5-fold splitting, training on 4 folds and evaluating on the held-out 5th fold, five separate times per model — so a manual train_test_split up front would just be redundant.

Section 3 · Step 2

Defining Four Candidate Models

Each model is wrapped in its own Pipeline with a StandardScaler step. That matters more than it looks: when cross_validate runs, the scaler gets refit from scratch on each fold's training portion only — exactly the same leakage-avoidance discipline from Lesson 12 Section 4, just applied automatically across all 5 folds instead of one manual split.

define_models.py

PYTHON

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.svm import SVR
from sklearn.neighbors import KNeighborsRegressor
from sklearn.ensemble import GradientBoostingRegressor

candidates = {
    "LinearRegression": Pipeline([
        ("scaler", StandardScaler()),
        ("model", LinearRegression())
    ]),
    "SVR (RBF kernel)": Pipeline([
        ("scaler", StandardScaler()),
        ("model", SVR(kernel="rbf", C=10, epsilon=0.1))
    ]),
    "KNeighborsRegressor": Pipeline([
        ("scaler", StandardScaler()),
        ("model", KNeighborsRegressor(n_neighbors=5))
    ]),
    "GradientBoostingRegressor": Pipeline([
        ("scaler", StandardScaler()),
        ("model", GradientBoostingRegressor(
            n_estimators=200, learning_rate=0.05, max_depth=3, random_state=42
        ))
    ]),
}

✨

GradientBoostingRegressor doesn't need scaling — but a shared pipeline shape keeps things clean

Same note as Lesson 12: tree-based models split on raw feature thresholds and are largely scale-insensitive. Every model here still gets the identical Pipeline([("scaler", ...), ("model", ...)]) shape purely so the comparison code in Section 4 can loop over all four candidates identically, with no special-casing.

Section 4 · Step 3

Cross-Validating All Four Models with cross_validate

Lesson 11 used cross_val_score for a single metric at a time. Its sibling, sklearn.model_selection.cross_validate, accepts a DICTIONARY of scorers and computes all of them per fold in one pass — exactly what's needed to get MAE, RMSE, and R² (Lesson 15) together.

cross_validate_all.py

PYTHON

from sklearn.model_selection import cross_validate

scoring = {
    "rmse": "neg_root_mean_squared_error",
    "mae": "neg_mean_absolute_error",
    "r2": "r2",
}

cv_results = {}
for name, pipeline in candidates.items():
    result = cross_validate(pipeline, X, y, cv=5, scoring=scoring)
    cv_results[name] = result
    print(f"{name}: fold RMSEs = {(-result['test_rmse']).round(0)}")

# LinearRegression: fold RMSEs = [29450. 31200. 28700. 30850. 29100.]
# SVR (RBF kernel): fold RMSEs = [34600. 33900. 35400. 34100. 33750.]
# KNeighborsRegressor: fold RMSEs = [32100. 31450. 32900. 31000. 31650.]
# GradientBoostingRegressor: fold RMSEs = [25100. 26300. 24700. 25850. 25200.]

⚠️

Scikit-learn's "neg_" scorers are negative on purpose

Scoring functions in scikit-learn follow a "higher is always better" convention, but RMSE and MAE are naturally "lower is better." The fix: scikit-learn negates them, so "neg_root_mean_squared_error" returns NEGATIVE RMSE values — the least-negative (closest to zero) is the best fold. That's exactly why the code above negates them back with a leading minus sign, -result["test_rmse"], before printing or averaging.

Section 5 · Step 4

Building the Comparison Table

Five fold scores per model, per metric, is a lot of numbers — collapse them into mean AND standard deviation, then assemble one small pandas DataFrame that's actually readable.

build_comparison_table.py

PYTHON

import pandas as pd

rows = []
for name, result in cv_results.items():
    rows.append({
        "Model": name,
        "RMSE (mean)": -result["test_rmse"].mean(),
        "RMSE (std)": result["test_rmse"].std(),
        "MAE (mean)": -result["test_mae"].mean(),
        "R2 (mean)": result["test_r2"].mean(),
    })

comparison_df = pd.DataFrame(rows).sort_values("RMSE (mean)").reset_index(drop=True)
print(comparison_df.round(0))

Illustrative 5-fold cross-validation results — a toy dataset, not a benchmark claim

Model	RMSE (mean)	RMSE (std)	MAE (mean)	R² (mean)
GradientBoostingRegressor	$25,430	$570	$18,760	0.906
LinearRegression	$29,860	$920	$22,410	0.869
KNeighborsRegressor	$31,820	$650	$24,050	0.851
SVR (RBF kernel)	$34,350	$580	$25,900	0.828

Illustrative mean CV RMSE by model — lower is better

GradientBoostingRegressor

$25,430

R2=0.91

LinearRegression

$29,860

R2=0.87

KNeighborsRegressor

$31,820

R2=0.85

SVR (RBF kernel)

$34,350

R2=0.83

✨

The standard deviation column matters as much as the mean

In this illustrative run, GradientBoostingRegressor has both the lowest mean RMSE AND a tight $570 standard deviation across its 5 folds — meaning it performed consistently well, not just well on average by luck of one easy fold. LinearRegression's wider $920 standard deviation signals more fold-to-fold variability, worth factoring into a real deployment decision alongside the raw mean.

⚠️

Untuned SVR often looks worse than it could be

SVR is highly sensitive to its C and epsilon hyperparameters — the values used here are reasonable defaults, not the result of tuning. A fair follow-up (see this lesson's challenge) would run GridSearchCV (Lesson 11) on SVR specifically before ruling it out entirely; it's very possible its ranking here would improve.

Section 6 · Step 5

Residual Analysis on the Winning Model

A low RMSE alone doesn't guarantee healthy errors (Lesson 15, Section 6) — so before recommending GradientBoostingRegressor, fit it once on a proper train/test split and look at its residuals directly.

residual_check_winner.py

PYTHON

from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

winner = candidates["GradientBoostingRegressor"]
winner.fit(X_train, y_train)
y_pred = winner.predict(X_test)
residuals = y_test - y_pred

fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].scatter(y_test, y_pred, alpha=0.5)
axes[0].plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], "r--")
axes[0].set_xlabel("Actual"); axes[0].set_ylabel("Predicted")
axes[0].set_title("Predicted vs Actual")

axes[1].scatter(y_pred, residuals, alpha=0.5)
axes[1].axhline(y=0, color="r", linestyle="--")
axes[1].set_xlabel("Predicted"); axes[1].set_ylabel("Residuals")
axes[1].set_title("Residuals vs Predicted")

plt.tight_layout(); plt.show()

Illustrative — the winner's residuals vs predicted

A reasonably random, flat cloud around zero — no strong funnel or curve visible, though spread widens very slightly at the high end.

✨

The residual plot backs up, rather than overturns, the metrics

If the winning model by RMSE had shown a strong funnel or curve here, that would be a real reason to pause before recommending it — a good RMSE hiding a badly-shaped error pattern is exactly the trap Lesson 15 warned about. In this illustrative run, the shape looks reasonably healthy, so the metrics and the residual plot AGREE — which is what you want to see before writing a recommendation.

Section 7 · Step 6

Writing the Recommendation

A comparison table isn't the deliverable — a decision is. Below is the kind of short, direct paragraph a real project report would end with. Treat the specific numbers as illustrative for this toy dataset, not a general claim that gradient boosting always beats these other three algorithms.

📄 Recommendation

Deploy GradientBoostingRegressor. Across 5-fold cross-validation it had the lowest mean RMSE ($25,430) and MAE ($18,760), the highest mean R² (0.906), and the tightest RMSE standard deviation ($570) of the four candidates — meaning it wasn't just lucky on one fold. Its residual plot on a held-out test split also showed a reasonably healthy, random spread with no strong funnel or curve, so the strong metrics aren't hiding a shape problem. LinearRegression is the runner-up and would be the fallback choice if model interpretability (reading its coefficients directly) mattered more than the roughly 15% RMSE improvement gradient boosting provides. KNeighborsRegressor and untuned SVR trail on every metric and aren't recommended as-is.

🏆

Winner: GradientBoostingRegressor

Best on RMSE, MAE, and R² simultaneously, with low fold-to-fold variance — the strongest, most consistent candidate in this comparison.

🥈

Runner-up: LinearRegression

Worse accuracy, but far more interpretable — a real trade-off worth weighing if stakeholders need to explain individual predictions.

⏱️

A factor the metrics alone don't show: training time

GradientBoostingRegressor with 200 estimators trains slower than LinearRegression or KNeighborsRegressor — usually a non-issue for a dataset this size, but worth checking at production scale.

🔁

Not the end of the road

This recommendation is based on DEFAULT or lightly-set hyperparameters for every model except the winner. Lesson 11's GridSearchCV on each candidate — this lesson's challenge — could shift these results further.

⚠️

"Best on this comparison" is not "best in general"

Exactly like Lesson 12's finding, this is a result on ONE synthetic dataset, with ONE set of hyperparameters per model, under 5-fold cross-validation. A different real dataset, different features, or tuned hyperparameters for the other three candidates could change the ranking. That's precisely why this project ran all four models honestly through the same pipeline instead of assuming a winner up front.

Section 8

The Complete Script, Start to Finish

Every step from this project, combined into one runnable comparison pipeline.

model_comparison_report.py — COMPLETE PROGRAM

PYTHON

import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.svm import SVR
from sklearn.neighbors import KNeighborsRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import cross_validate, train_test_split

# 1. Load and preprocess (reusing Lesson 12's steps)
df = pd.read_csv("house_prices.csv")
df["bedrooms"] = df["bedrooms"].fillna(df["bedrooms"].median())
df_encoded = pd.get_dummies(df, columns=["city"], drop_first=True)
X = df_encoded.drop(columns=["price"])
y = df_encoded["price"]

# 2. Define four candidate pipelines
candidates = {
    "LinearRegression": Pipeline([("scaler", StandardScaler()), ("model", LinearRegression())]),
    "SVR (RBF kernel)": Pipeline([("scaler", StandardScaler()), ("model", SVR(kernel="rbf", C=10, epsilon=0.1))]),
    "KNeighborsRegressor": Pipeline([("scaler", StandardScaler()), ("model", KNeighborsRegressor(n_neighbors=5))]),
    "GradientBoostingRegressor": Pipeline([
        ("scaler", StandardScaler()),
        ("model", GradientBoostingRegressor(n_estimators=200, learning_rate=0.05, max_depth=3, random_state=42))
    ]),
}

# 3. Cross-validate all four on MAE, RMSE, R2 together
scoring = {"rmse": "neg_root_mean_squared_error", "mae": "neg_mean_absolute_error", "r2": "r2"}
rows = []
for name, pipeline in candidates.items():
    result = cross_validate(pipeline, X, y, cv=5, scoring=scoring)
    rows.append({
        "Model": name,
        "RMSE (mean)": -result["test_rmse"].mean(),
        "RMSE (std)": result["test_rmse"].std(),
        "MAE (mean)": -result["test_mae"].mean(),
        "R2 (mean)": result["test_r2"].mean(),
    })

# 4. Rank the comparison table
comparison_df = pd.DataFrame(rows).sort_values("RMSE (mean)").reset_index(drop=True)
print(comparison_df.round(0))

# 5. Residual-check the winner on a held-out split
best_name = comparison_df.iloc[0]["Model"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
winner = candidates[best_name]
winner.fit(X_train, y_train)
residuals = y_test - winner.predict(X_test)
print(f"Winner: {best_name}, residual mean: {residuals.mean():.1f}, residual std: {residuals.std():.1f}")

🧩 Knowledge Check — Lesson 16

4 questions on cross-validated model comparison before you finish Section 3.

1. Why use cross_validate with cv=5 to compare these four models, instead of one train_test_split like Lesson 12 used?

2. Why is StandardScaler wrapped inside each model's Pipeline, rather than scaling X once before calling cross_validate?

3. In this project's illustrative cross-validation results, which model had both the lowest mean RMSE and the lowest mean MAE?

4. Which statement best describes how this project decided what to recommend deploying?

💪

Try It Yourself — Lesson 16

Extend the model comparison project · Advanced Level

Use the candidates dictionary and comparison_df from Sections 3–5 as your starting point for each task below.

Task 1: Tune the SVR before ruling it out 🎛️

Using Lesson 11's GridSearchCV, search over C in [1, 10, 100] and epsilon in [0.05, 0.1, 0.5] for the SVR pipeline, with cv=5 and scoring="neg_root_mean_squared_error". Does its tuned RMSE close the gap with GradientBoostingRegressor, or even overtake it?

Task 2: Add a fifth contender 🥊

Add RandomForestRegressor (from sklearn.ensemble) as a fifth pipeline in candidates, using similar settings to the GradientBoostingRegressor (n_estimators=200, random_state=42). Rerun Section 4's cross-validation loop and Section 5's comparison table with all five models included.

Task 3: Residual-check the runner-up too 📊

Repeat Section 6's residual analysis, but for LinearRegression instead of the winner. Compare its residuals-vs-predicted plot to the winner's — does it show more of a pattern (heteroscedasticity or non-linearity, from Lesson 15)? Write 2–3 sentences on whether that changes or reinforces this lesson's recommendation.

💡 Show hints if you're stuck

Task 1: GridSearchCV(candidates["SVR (RBF kernel)"], param_grid={"model__C": [1,10,100], "model__epsilon": [0.05,0.1,0.5]}, cv=5, scoring="neg_root_mean_squared_error") — note the model__ prefix needed to reach inside a Pipeline's named step.
Task 2: from sklearn.ensemble import RandomForestRegressor — it takes the exact same Pipeline shape as the other tree-based model already in candidates.
Task 3: If LinearRegression's residuals show a visible curve while the winner's don't, that's independent evidence GradientBoostingRegressor is capturing a real non-linear relationship LinearRegression misses — strengthening, not weakening, the original recommendation.

Finished the capstone project?

Mark it complete to track your progress.

🎉

Section 3 Complete — Capstone Project Done!

You've now cross-validated four different regression algorithms honestly, built a real comparison table with MAE/RMSE/R², checked a winning model's residuals before trusting its metrics, and written an actual deployment recommendation. That's every core skill from Lessons 13–15, combined into one real workflow. Section 4 is next — leaving supervised learning behind for clustering and unsupervised methods.

Module 16 of 24 Section 3 — Model Evaluation Mastery