📐 Section 2 · Supervised Learning 🟡 Intermediate MODULE 06

Linear Regression — Theory & Math

⏱️ 26 min read

📖 First Real Algorithm

🧩 4 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 214%

🎯 What you'll learn: Welcome to Section 2 — real algorithms, starting with the simplest and most foundational one. You'll see linear regression from the ground up: the hypothesis function that makes a prediction, the cost function that measures how wrong it is, gradient descent as the engine that improves it, and finally how all of that collapses into a single .fit() call in scikit-learn.

Section 1

The Hypothesis Function

Linear regression starts from a simple assumption: the target value is roughly a weighted sum of the input features, plus a constant. For one feature (simple linear regression), the model's prediction is:

Simple Linear Regression — One Feature

ŷ = w·x + b

ŷ ("y-hat") is the predicted value, x is the input feature, w is the weight (slope), b is the bias (intercept).

Geometrically, this is just the equation of a straight line — w controls how steep it is, b controls where it crosses the y-axis. "Fitting" the model means finding the specific values of w and b that make this line track the data as closely as possible.

Real datasets almost always have more than one feature. Multiple linear regression generalizes the same idea — one weight per feature, all summed together:

Multiple Linear Regression — Many Features

ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b

Each feature xᵢ gets its own learned weight wᵢ. In vector form: ŷ = w · x + b

📝

"Linear" refers to the parameters, not necessarily a straight line in feature space

The model is linear in w and b — each feature contributes additively, weighted by a single number. This is what makes linear regression fast to train and easy to interpret, but it also means it can't natively capture curves or interactions between features without extra work (like adding polynomial features).

Section 2

The Cost Function — Mean Squared Error

Before you can find good values for w and b, you need a way to measure how bad any particular choice of them is. That's the job of a cost function (also called a loss function). Linear regression's standard cost function is Mean Squared Error (MSE): the average of the squared differences between each true value and each prediction.

Mean Squared Error

MSE = (1/n) · Σ (yᵢ − ŷᵢ)²

n = number of training rows, yᵢ = the true value for row i, ŷᵢ = the model's prediction for row i.

➖

Why subtract?

(yᵢ − ŷᵢ) is the "residual" — how far off a single prediction was, in either direction.

✖️

Why square?

Squaring makes every error positive (so overshooting and undershooting don't cancel out) and punishes large errors disproportionately more than small ones.

➗

Why average?

Dividing by n turns the total squared error into a per-row average, so the cost doesn't just grow because the dataset has more rows.

🎯

The goal

"Training" the model means searching for the w and b that make MSE as small as possible — that's the entire optimization problem.

✨

MSE vs. RMSE

You'll often see RMSE (Root Mean Squared Error) reported instead of raw MSE — it's just the square root of MSE. Taking the square root brings the error back into the same units as the original target (e.g. dollars, not dollars-squared), which makes it much easier to interpret directly.

Section 3

Gradient Descent — How the Model Actually Learns

MSE gives a single number describing how bad the current w and b are. Gradient descent is the algorithm that iteratively nudges w and b in the direction that makes MSE smaller, repeating until it stops improving much.

Start somewhere

Initialize w and b to some starting values — often small random numbers, or even zero.

Measure the slope of the cost

Compute the gradient — the direction in which MSE increases fastest, with respect to w and with respect to b.

Step in the opposite direction

Since the gradient points toward higher cost, move w and b a small step in the OPPOSITE direction — toward lower cost.

Repeat

Do this over and over. Each pass, the cost should get a little lower, until further steps stop helping much (convergence).

The Gradient Descent Update Rule

w = w − α · ∂MSE/∂w

α (alpha) is the learning rate — how big each step is. The same update rule applies to b.

🥾

The hiker-in-fog analogy

Picture standing on a hilly landscape in thick fog, trying to reach the lowest point (minimum cost) by feel alone. You can't see the whole landscape, but you CAN feel which direction is downhill right where you're standing — that's the gradient. Gradient descent takes a small step downhill, feels the new slope, takes another small step, and repeats. Take steps too small (low learning rate) and you'll get there eventually but very slowly; take steps too large (high learning rate) and you risk overshooting the bottom entirely, or bouncing around without ever settling.

🐢

Learning rate too small

Training converges, but takes a very long time — tiny steps toward the minimum.

🚀

Learning rate too large

Steps overshoot the minimum, and the cost can oscillate wildly or even diverge (grow instead of shrink).

📉 Illustrative pattern — cost vs. gradient descent iterations

Picture the x-axis as training iterations and the y-axis as MSE. With a well-chosen learning rate, the curve drops steeply at first, then flattens out as it approaches the minimum — the classic "elbow" shape of a converging cost curve. A learning rate that's too high would instead show a jagged, non-decreasing (or even increasing) line.

📝

Gradient descent isn't the only way to solve linear regression

For linear regression specifically, there's also a closed-form solution called the normal equation that computes the optimal w directly with matrix algebra, without any iteration. Gradient descent matters more broadly because it scales to huge datasets and generalizes to models (like neural networks) that have no such closed-form shortcut — which is exactly why it's introduced here rather than skipped.

Section 4

From Math to One Line: sklearn.linear_model.LinearRegression

Every bit of math above is exactly what happens inside scikit-learn's LinearRegression when you call .fit(). It runs an optimization (scikit-learn actually uses a fast linear-algebra solver rather than manual gradient descent under the hood, but the objective it's minimizing is the same MSE) and hands you back the learned w and b as .coef_ and .intercept_.

linear_regression.py

PYTHON

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# This one call finds the w and b that minimize MSE on the training data
model = LinearRegression()
model.fit(X_train, y_train)

print("Learned weights (one per feature):", model.coef_)
print("Learned bias / intercept:", model.intercept_)

y_pred = model.predict(X_test)
print("Test MSE:", mean_squared_error(y_test, y_pred))
print("Test R²:", r2_score(y_test, y_pred))

⚖️

.coef_

The learned wᵢ values, one per input feature — exactly the "w" from the hypothesis function.

🔀

.intercept_

The learned b — exactly the bias term from the hypothesis function.

📉

mean_squared_error()

Computes the exact MSE formula from Section 2, but on the TEST set — how well the fitted line generalizes to unseen rows.

📈

r2_score()

R² (R-squared) measures the proportion of variance in y explained by the model, from 0 (no better than predicting the mean) to 1 (perfect fit).

✨

Every coefficient is directly interpretable

Because the model is a simple weighted sum, coef_[i] tells you exactly how much ŷ changes when feature i increases by 1 unit, holding all other features fixed. This interpretability is one of linear regression's biggest practical advantages over more complex models later in this section — you can literally read the equation.

The illustrative numbers below show what model.coef_ and model.intercept_ might look like for a toy house-price example with two features — this is a demonstration of the shape of the output, not a real fitted model:

output (illustrative)

OUTPUT

# Learned weights (one per feature): [142.3  8210.7]
#   -> +1 sqft is worth ~$142 more, +1 bedroom is worth ~$8,211 more
# Learned bias / intercept: 15320.5
#   -> baseline value when all features are 0 (rarely meaningful on its own)

Section 5

Lesson Summary

✅The hypothesis function ŷ = w·x + b (or ŷ = w₁x₁ + ... + wₙxₙ + b) is how linear regression makes predictions.

✅The MSE cost function measures how wrong those predictions are, averaged over all rows.

✅Gradient descent iteratively adjusts w and b downhill on the cost surface, controlled by a learning rate.

✅LinearRegression().fit() in scikit-learn solves that exact optimization, exposing the results as .coef_ and .intercept_.

🧩 Knowledge Check — Lesson 6

4 questions on the math and mechanics of linear regression.

1. In ŷ = w·x + b, what does b represent?

2. Why does Mean Squared Error square the residuals instead of just averaging them directly?

3. What does a learning rate that's set too HIGH tend to cause in gradient descent?

4. After calling model.fit(X_train, y_train) on a LinearRegression, where do you find the learned weights?

💪

Try It Yourself — Lesson 6

Connect the math to the code · Intermediate Level

These tasks connect the formulas from Sections 1–3 to the real scikit-learn code in Section 4.

Task 1: Compute MSE by hand 🧮

For three predictions — true values [10, 20, 30] and predicted values [12, 18, 33] — compute the residuals, square each one, and average them to get MSE by hand. Then verify your answer using sklearn.metrics.mean_squared_error([10,20,30], [12,18,33]).

Task 2: Fit and read the coefficients 📐

Using the code sample in Section 4, fit a LinearRegression on any dataset with at least 2 features. Print model.coef_ and model.intercept_, then write one sentence per feature explaining what its coefficient means in plain English.

Task 3: Reason about learning rate 🎛️

Without running any code, predict what would happen to gradient descent's convergence if the learning rate α were set to 10 instead of a typical small value like 0.01. Explain your reasoning using the hiker-in-fog analogy from Section 3.

💡 Show hints if you're stuck

Task 1: Residuals are -2, 2, -3. Squared: 4, 4, 9. Average: (4+4+9)/3 = 5.67.
Task 2: A positive coefficient means increasing that feature increases the predicted value; a negative one means the opposite — the size of the number tells you how much, per one unit of that feature.
Task 3: α = 10 is a huge step size for most problems — it would very likely overshoot the minimum repeatedly, potentially making the cost grow instead of shrink (diverge), like a hiker taking such large strides they leap right over the valley floor.

Finished this lesson?

Mark it complete to track your progress.

Module 06 of 24 Section 2 — Supervised Learning Algorithms