🧠 Section 1 · Foundations 🟡 Intermediate MODULE 02

The Machine Learning Pipeline

⏱️ 22 min read

📖 Workflow & Process

🧩 4 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 140%

🎯 What you'll learn: Every ML project — no matter which algorithm ends up doing the work — follows the same repeatable shape: frame the problem, get the data, clean it, engineer features, split it, pick and train a model, evaluate it, tune it, and eventually deploy it. This lesson walks through all ten stages, then shows how scikit-learn's Pipeline class lets you chain the preprocessing and modeling steps into a single reusable object.

Section 1

Why Think in Terms of a Pipeline?

It's tempting to think of "doing machine learning" as the moment you call .fit() on a model. In practice, that single line sits in the middle of a much longer process — and most of the time in a real project goes into the steps before and after it. Treating the whole thing as a pipeline — a fixed sequence of stages that data flows through — makes projects reproducible, makes it obvious where a bug or a leak of information could have crept in, and makes it possible to swap one stage (a different model, a different scaler) without rebuilding everything else.

📝

This lesson previews the shape — later lessons fill in the details

Every stage below gets its own deeper treatment later in the course: preprocessing in Lesson 3, splitting and cross-validation in Lesson 4, overfitting-aware model selection in Lesson 5, and specific algorithms from Section 2 onward. For now, focus on the order of operations and why each stage exists.

Section 2

The Ten Stages

These stages run roughly in order, though in practice you'll loop back — a disappointing evaluation often sends you back to feature engineering or even problem framing.

Problem framing

Decide exactly what you're predicting and why. Is this regression or classification? What does "good enough" look like? A vague goal ("predict churn") should become a precise one ("predict, for each active customer, the probability they cancel within 30 days").

Data collection

Gather the raw data the problem needs — from a database, an API, log files, or existing CSVs. The quality and relevance of this data caps how good any model downstream can possibly be.

Data cleaning

Handle missing values, fix inconsistent formatting, remove duplicates, and deal with obvious errors. Nothing downstream can be trusted until this stage is done honestly.

Feature engineering

Transform raw columns into inputs a model can use well — encoding categories, scaling numbers, combining or extracting new signal (like turning a timestamp into "day of week"). Lesson 3 covers encoding and scaling in depth.

Train/test split

Hold out a portion of the data that the model never sees during training, so you have an honest way to measure how it performs on new data later. Lesson 4 covers this and cross-validation in depth.

Model selection

Choose one or more candidate algorithms appropriate for the problem type — the algorithm families covered in Sections 2–4 of this course.

Training

Call .fit() on the training data. This is the step where the algorithm actually searches for parameters — everything before it exists to make this step meaningful.

Evaluation

Measure performance on the held-out test data with a metric appropriate to the problem — never on the training data alone.

Tuning

Adjust hyperparameters (settings the algorithm doesn't learn on its own, like tree depth or regularization strength) and repeat training/evaluation to improve results, ideally using cross-validation rather than the test set directly.

Deployment

Put the trained model where it can actually make predictions on new, real-world data — behind an API, inside a batch job, or embedded in an application — and monitor it, since real-world data can drift away from what the model was trained on.

The ML Pipeline at a Glance

Frame Problem

→

Collect Data

→

Clean Data

→

Engineer Features

→

Train/Test Split

→

Select & Train Model

→

Evaluate & Tune

→

Deploy

⚠️

Order matters more than it looks

Fitting a scaler or an encoder on the FULL dataset before splitting into train/test — instead of fitting only on the training set — quietly leaks information from the test set into training. Lesson 3 covers this "fit-on-train-only" rule in detail; it's one of the most common mistakes in real ML code.

Section 3

scikit-learn's Pipeline Class

scikit-learn provides a Pipeline class that chains preprocessing steps and a final model into a single object. Instead of manually calling .fit_transform() on a scaler, then .fit() on a model, and remembering to repeat the exact same transform steps at prediction time, you build the sequence once and call .fit() and .predict() on the whole pipeline.

pipeline_skeleton.py

PYTHON

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# X, y already loaded as a NumPy array / pandas DataFrame and Series
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Each step is a (name, transformer_or_estimator) tuple, in order
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('model', LogisticRegression())
])

# One call trains the scaler AND the model, in the correct order
pipe.fit(X_train, y_train)

# Predicting automatically applies the SAME fitted scaler before the model
predictions = pipe.predict(X_test)
accuracy = pipe.score(X_test, y_test)
print("Test accuracy:", accuracy)

✨

Why bother with Pipeline instead of doing it manually?

Two big reasons. First, it removes a whole class of bugs: it's easy to accidentally scale the test set with a scaler you re-fit on the test set, or forget to apply a transform at prediction time — Pipeline makes that mistake structurally impossible. Second, it makes cross-validation (Lesson 4) and hyperparameter search safe by default, because each fold refits the scaler on only that fold's training data.

Every step except the last must be a transformer — something with both .fit() and .transform(), like StandardScaler or OneHotEncoder (covered next lesson). The last step is the estimator — the actual model, like LogisticRegression or DecisionTreeClassifier — which needs .fit() and .predict().

Section 4

Lesson Summary

✅Every ML project follows a repeatable ten-stage pipeline: frame → collect → clean → engineer → split → select → train → evaluate → tune → deploy.

✅Most real project time goes into the stages before and after .fit(), not the training call itself.

✅The train/test split must happen before fitting any preprocessing step, to avoid leaking test information into training.

✅scikit-learn's Pipeline class chains transformers and a final estimator into one object, calling .fit() and .predict() in the right order automatically.

🧩 Knowledge Check — Lesson 2

4 questions on the ML pipeline and scikit-learn's Pipeline class.

1. Which pipeline stage comes immediately BEFORE training the model?

2. Why is it a mistake to fit a StandardScaler on the full dataset before splitting into train and test?

3. In a scikit-learn Pipeline([('scaler', StandardScaler()), ('model', LogisticRegression())]), what must every step EXCEPT the last one implement?

4. What does calling pipe.predict(X_test) do, when pipe is a fitted Pipeline containing a scaler and a model?

💪

Try It Yourself — Lesson 2

Trace the pipeline end-to-end · Intermediate Level

These tasks are about tracing the pipeline shape and getting comfortable writing a basic Pipeline.

Task 1: Diagnose a broken project 🔍

A teammate says their model scores 98% during evaluation but performs terribly once deployed. They mention they scaled their entire dataset with StandardScaler before splitting into train and test. In 2–3 sentences, explain what likely went wrong and which pipeline stage should be reordered.

Task 2: Build a two-step Pipeline 🧱

Using the code sample in Section 3 as a reference, write a Pipeline that chains a MinMaxScaler (instead of StandardScaler) with a LogisticRegression model, fit it on X_train/y_train, and print its .score() on the test set.

Task 3: Map your own project to the ten stages 🗺️

Think of any dataset or problem you're curious about (sports results, weather, a hobby you track data for). Write one sentence for each of the ten pipeline stages describing what that stage would concretely involve for your chosen problem.

💡 Show hints if you're stuck

Task 1: Fitting the scaler on the full dataset lets statistics from the test rows (their mean and standard deviation) influence the scaling applied to training rows — a data leak. The fix is to split first, then fit the scaler only on X_train.
Task 2: Pipeline([('scaler', MinMaxScaler()), ('model', LogisticRegression())]), then pipe.fit(X_train, y_train) and print(pipe.score(X_test, y_test)).
Task 3: It's fine if some stages feel thin (e.g. "problem framing: predict whether my weekly 5K time will improve") — the goal is recognizing the shape applies broadly, not writing a perfect plan.

Finished this lesson?

Mark it complete to track your progress.

Module 02 of 24 Section 1 — What is Machine Learning?