🤖 Section 5 · Machine Learning 🟡 Intermediate MODULE 25

Linear & Logistic Regression

⏱️ 55 min

📖 Your First Two Models

🧩 3 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 529%

🎯 Your first two real models. Lesson 22 fit a line of best fit with scipy.stats.linregress purely for statistical analysis. This lesson fits the exact same kind of line — but through scikit-learn's LinearRegression class, following the fit/predict workflow from Lesson 24, so it can actually be evaluated and reused for prediction. Then we flip to classification: LogisticRegression predicts categories, not numbers, using a clever S-shaped function called the sigmoid to turn any input into a probability between 0 and 1.

Section 1

Linear Regression, the Scikit-Learn Way

In scikit-learn, every model is a class with the same shape: create an instance, call .fit(X, y) to train it, then call .predict(X_new) to get predictions. LinearRegression is the simplest possible example of this pattern.

linear_regression_sklearn.py

PYTHON

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Hours studied (X must be 2D) vs. exam score (y is 1D) — same data as Lesson 22
X = np.array([[1], [2], [2], [3], [4], [4], [5], [6], [7], [8]])
y = np.array([52, 58, 55, 64, 68, 70, 75, 80, 85, 92])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Create, then fit, the model
model = LinearRegression()
model.fit(X_train, y_train)

# The learned parameters live on the fitted model
print(f"Slope (coef_):     {model.coef_[0]:.3f}")
print(f"Intercept:         {model.intercept_:.3f}")

# Predict on the held-out test rows
y_pred = model.predict(X_test)
print("Predictions:", y_pred.round(1))
print("Actual:     ", y_test)

# Predict for a brand-new student who studies 6.5 hours
new_student = np.array([[6.5]])
print(f"Predicted score at 6.5 hrs: {model.predict(new_student)[0]:.1f}")

📝

X is always 2D, even with a single feature

scikit-learn always expects X shaped as (n_samples, n_features) — a 2D array, even when there's only one feature. That's why X above is a list of one-item lists, [[1],[2],...], rather than a flat [1,2,...]. y, the target, stays 1D. Passing a flat 1D X raises a ValueError in most scikit-learn versions.

✨

Same math as Lesson 22, different tool

model.coef_ and model.intercept_ are the scikit-learn equivalents of result.slope and result.intercept from scipy.stats.linregress() in Lesson 22 — both fit the exact same "line of best fit" via ordinary least squares. The scikit-learn version is preferred once you want the reusable .fit()/.predict() workflow, multiple input features, or to plug the model into the rest of the scikit-learn ecosystem (like cross_val_score in Lesson 27).

Section 2

Evaluating a Regression Model — MSE, RMSE, R²

A regression prediction is a number, so "how wrong was it" is also measured in numbers — but there are a few different ways to summarize that, each with a slightly different meaning.

Mean Squared Error (MSE)

MSE = (1/n) · Σ(yᵢ − ŷᵢ)²

The average of the squared differences between actual (yᵢ) and predicted (ŷᵢ) values. Squaring punishes big misses more than small ones, but it also means MSE is in squared units — hard to interpret directly.

📐

RMSE — same units as y

The square root of MSE. If y is exam scores, RMSE is also in "points" — much easier to interpret than MSE's squared units.

📊

R² — proportion of variance explained

Same r² concept from Lesson 22, 1.0 is a perfect fit, 0.0 means the model does no better than always predicting the mean of y.

⬇️

Lower MSE/RMSE = better

These are error metrics — smaller is better, and 0 would mean every prediction was exactly right.

⬆️

Higher R² = better

Unlike MSE/RMSE, R² is a "goodness" score — higher (closer to 1.0) is better, and it can even go negative for a model worse than just guessing the mean.

evaluate_regression.py

PYTHON

from sklearn.metrics import mean_squared_error, r2_score

# y_pred and y_test from Section 1's train/test split
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)          # take the square root manually to get RMSE
r2 = r2_score(y_test, y_pred)

print(f"MSE:  {mse:.2f}")
print(f"RMSE: {rmse:.2f}")
print(f"R²:   {r2:.4f}")

# .score() on a fitted regressor is a shortcut that returns R² directly
print(f"model.score() R²: {model.score(X_test, y_test):.4f}")

⚠️

On such a tiny illustrative dataset, don't over-read the exact numbers

With only 10 total rows and a 2-row test set, these particular MSE/RMSE/R² values are just illustrative of how the functions are called — they're far too noisy to represent real model quality. Real projects evaluate on hundreds or thousands of test rows, and lean on cross-validation (Lesson 27) rather than a single train/test split.

Section 3

Logistic Regression — Predicting Categories

Despite the name, logistic regression is a classification algorithm, not a regression one — it predicts a category (most often, one of two classes) rather than a continuous number. It works by fitting a linear combination of the inputs, then squashing that value into a probability with the sigmoid function.

The Sigmoid Function

σ(z) = 1 / (1 + e^(−z))

No matter what raw number z comes out of the linear part of the model — from a huge negative number to a huge positive one — σ(z) always squashes it into the range (0, 1), which can then be read as a probability.

Compute a linear score

Just like linear regression, the model computes z = w₁x₁ + w₂x₂ + ... + b from the input features and learned weights.

Squash with sigmoid

z is passed through σ(z) to produce a probability between 0 and 1 — e.g. "78% probability this is class 1."

Apply a threshold

By default, scikit-learn predicts class 1 if the probability is ≥ 0.5, and class 0 otherwise. That 0.5 cutoff is called the decision boundary.

logistic_regression.py

PYTHON

from sklearn.linear_model import LogisticRegression

# Hours studied vs. pass (1) / fail (0) — a small, illustrative, imperfectly-separable dataset
X_clf = np.array([[1], [2], [2.5], [3], [4], [5], [5.5], [6], [7], [8]])
y_clf = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])

X_train, X_test, y_train, y_test = train_test_split(
    X_clf, y_clf, test_size=0.3, random_state=42
)

clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)

# .predict() returns the class (0 or 1); .predict_proba() returns both class probabilities
print("Predicted classes:", clf.predict(X_test))
print("Actual classes:   ", y_test)
print("Probabilities [P(fail), P(pass)]:\n", clf.predict_proba(X_test).round(3))

# Accuracy on the test set (fraction of correct predictions)
print(f"Accuracy: {clf.score(X_test, y_test):.2f}")

# Where does this model draw its decision boundary, in hours studied?
# Solve coef_*x + intercept_ = 0  →  x = -intercept_ / coef_
boundary = -clf.intercept_[0] / clf.coef_[0][0]
print(f"Decision boundary at ~{boundary:.2f} hours studied")

✨

predict() vs. predict_proba()

.predict() is just a convenience wrapper that applies the 0.5 threshold to .predict_proba() for you. Reaching for .predict_proba() directly is worth it whenever "how confident was the model" matters more than a flat yes/no — e.g. flagging only the highest-confidence fraud predictions for manual review.

📝

max_iter — a practical detail, not a modeling choice

LogisticRegression finds its weights with an iterative optimizer, and the default iteration cap (100) is sometimes too low to fully converge, which raises a ConvergenceWarning without necessarily being wrong. Passing max_iter=1000 gives the optimizer more room to finish cleanly — it's a numerical safety margin, not something that changes what the model is doing conceptually.

Section 4

Linear vs. Logistic — Choosing the Right One

Both models fit a straight-line-shaped relationship between features and target, and both are fast, interpretable, and a sensible first model to try — the difference is entirely about what kind of target you're predicting.

📈

LinearRegression

Target is continuous — price, temperature, exam score. Output can be any real number.

🏷️

LogisticRegression

Target is categorical — most commonly binary (0/1). Output is a probability between 0 and 1, then a predicted class.

📉

LinearRegression loss

Fit by minimizing squared error between predicted and actual numbers — the same least-squares idea from Lesson 22.

🎲

LogisticRegression loss

Fit by maximizing the likelihood of the observed classes given the predicted probabilities — a different objective, better suited to categorical outcomes.

⚠️

Never use LinearRegression to predict a 0/1 category

It's technically possible to fit a LinearRegression on a 0/1 target, but its raw output can land anywhere on the number line — including negative values or values above 1 — which makes no sense as a probability. LogisticRegression's sigmoid squashing is specifically designed to avoid this problem.

🧩 Knowledge Check — Lesson 25

3 questions on linear and logistic regression before you move on.

1. After calling model.fit(X_train, y_train) on a LinearRegression instance, where do the learned slope and intercept live?

2. What does the sigmoid function do inside LogisticRegression?

3. Why is LinearRegression the wrong choice for predicting a "spam / not spam" label directly?

💪

Try It Yourself — Lesson 25

Fit both model types · Intermediate Level

Use this illustrative dataset relating advertising spend (in thousands of PKR) to weekly units sold: ad_spend = [[10],[15],[20],[25],[30],[35],[40],[45]], units_sold = [102,118,135,149,168,180,195,210].

Task 1: Fit a LinearRegression 📈

Split the data with train_test_split(ad_spend, units_sold, test_size=0.25, random_state=1), fit a LinearRegression, and print model.coef_ and model.intercept_. In plain English, what does the coefficient represent?

Task 2: Evaluate it 📐

Predict on the test set, then compute MSE, RMSE (via np.sqrt), and R² with mean_squared_error and r2_score from sklearn.metrics.

Task 3: Turn it into a classification problem 🏷️

Create a new binary target: high_demand = [1 if u >= 150 else 0 for u in units_sold]. Fit a LogisticRegression on ad_spend to predict high_demand, and print .predict_proba() for an ad spend of 28 (thousand PKR) using model.predict_proba([[28]]).

💡 Show hints if you're stuck

Task 1: model.coef_[0] is the predicted increase in units_sold per extra 1,000 PKR of ad spend.
Task 2: mse = mean_squared_error(y_test, y_pred); rmse = np.sqrt(mse); r2 = r2_score(y_test, y_pred)
Task 3: Remember ad_spend is already a list of one-item lists, so it's already 2D and ready to pass straight into LogisticRegression().fit().

Finished this lesson?

Mark it complete to track your progress.

Module 25 of 30 Section 5 — Machine Learning with Scikit-Learn

Linear & Logistic Regression

Linear Regression, the Scikit-Learn Way

Evaluating a Regression Model — MSE, RMSE, R²

Logistic Regression — Predicting Categories

Linear vs. Logistic — Choosing the Right One

Lesson 25 Complete!