Regression Metrics & Residual Analysis

One Number Is Never the Whole Story

Lesson 12's capstone used RMSE and R² to compare LinearRegression against GradientBoostingRegressor — but only briefly explained what those numbers actually meant. This lesson slows down and covers the full regression-metrics toolbox properly: four metrics that each answer a slightly different question about the same set of prediction errors.

Every metric in this lesson starts from the same raw ingredient: the residual — the gap between what actually happened and what the model predicted.

MAE, MSE, RMSE, and R² are all different ways of summarizing a whole array of residuals — one per test-set row — into a single number. The differences between them come down to HOW they combine those residuals.

Mean Absolute Error (MAE) — Robust to Outliers

MAE takes the absolute value of every residual (so a $10,000 over-prediction and a $10,000 under-prediction count equally), then averages them. It answers a simple, very interpretable question: "on average, how many dollars off is a typical prediction?"

from sklearn.metrics import mean_absolute_error

mae = mean_absolute_error(y_test, y_pred)
print(f"MAE: ${mae:,.0f}")
# MAE: $22,410   -> on average, predictions are off by about $22,410

Mean Squared Error (MSE) & RMSE — Punishing Big Misses Harder

MSE squares every residual before averaging. Squaring does two things at once: it makes every error positive (like abs() did for MAE), and it makes LARGE errors count disproportionately more than small ones.

import numpy as np
from sklearn.metrics import mean_squared_error

mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)

print(f"MSE:  {mse:,.0f}")
print(f"RMSE: ${rmse:,.0f}")
# MSE:  1,142,650,000
# RMSE: $33,803

Notice RMSE ($33,803) came out higher than MAE ($22,410) on the exact same predictions. That gap is not a coincidence — it happens whenever the errors aren't all the same size, and it grows larger the more a few big misses stand out from the rest.

R² (Coefficient of Determination) — Proportion of Variance Explained

MAE and RMSE tell you the error in absolute units — but "$22,410 off" is hard to judge without context. Is that good? It depends entirely on whether the target ranges from $100,000 to $800,000, or from $20,000 to $30,000. R² instead expresses the model's performance RELATIVE to the simplest possible baseline: just predicting the mean every time.

from sklearn.metrics import r2_score

r2 = r2_score(y_test, y_pred)
print(f"R2: {r2:.3f}")
# R2: 0.884   -> the model explains about 88.4% of price's variance

Computing All Four Together

In practice, a real project prints all four side by side rather than picking just one — each one flags a different kind of problem.

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

def regression_report(y_true, y_pred, name="model"):
    mae = mean_absolute_error(y_true, y_pred)
    mse = mean_squared_error(y_true, y_pred)
    rmse = np.sqrt(mse)
    r2 = r2_score(y_true, y_pred)
    print(f"{name} -> MAE=${mae:,.0f}  RMSE=${rmse:,.0f}  R2={r2:.3f}")
    return {"mae": mae, "rmse": rmse, "r2": r2}

regression_report(y_test, y_pred, "LinearRegression")
# LinearRegression -> MAE=$22,410  RMSE=$33,803  R2=0.884

Residual Plots — Seeing What the Metrics Can't

MAE, RMSE, and R² each compress an entire array of residuals into one number — which is exactly their weakness. Two very different-looking sets of errors can produce the identical RMSE. A residual plot shows every individual error, so patterns invisible to any single metric become visible at a glance.

Two residual plots are standard, and they answer different questions:

import matplotlib.pyplot as plt

y_pred = model.predict(X_test)
residuals = y_test - y_pred

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# Plot 1: Predicted vs Actual — points should hug the diagonal
axes[0].scatter(y_test, y_pred, alpha=0.5)
axes[0].plot(
    [y_test.min(), y_test.max()],
    [y_test.min(), y_test.max()],
    "r--", label="Perfect prediction"
)
axes[0].set_xlabel("Actual")
axes[0].set_ylabel("Predicted")
axes[0].set_title("Predicted vs Actual")
axes[0].legend()

# Plot 2: Residuals vs Predicted — should be a random cloud around 0
axes[1].scatter(y_pred, residuals, alpha=0.5)
axes[1].axhline(y=0, color="r", linestyle="--")
axes[1].set_xlabel("Predicted")
axes[1].set_ylabel("Residuals")
axes[1].set_title("Residuals vs Predicted")

plt.tight_layout()
plt.show()

Lesson Summary