Model Monitoring & Drift Detection

Why a Deployed Model Can Degrade Over Time

A trained model is a snapshot. It learned patterns from a specific training dataset, collected at a specific point in time, and everything it does — every prediction from Lesson 22's /predict endpoint — is really just applying that frozen snapshot of patterns to new, incoming data. The problem is simple to state and easy to forget in practice: the real world keeps changing, and the model does not — not unless someone deliberately retrains it.

Two related but distinct things can go wrong, and telling them apart is the core of this lesson: the DATA a model sees can change (data drift), or the underlying RELATIONSHIP between that data and the correct answer can change (concept drift).

Data Drift vs. Concept Drift

Both describe a live model seeing something different from what it was trained on — the distinction is WHAT changed.

Data Drift (a.k.a. feature or input drift)

The distribution of the INPUT features shifts over time, even though the underlying relationship between those features and the correct answer stays the same. The model itself isn't wrong about the pattern — it's just being asked about a part of the "space" it never really saw much of during training.

Concept Drift

The relationship between the input features (X) and the correct answer (y) itself changes — the same input that used to mean one thing now means something different. This is a deeper problem than data drift, because no amount of the model just "seeing more of the same kind of data" fixes it; the pattern it learned is genuinely no longer accurate.

Simple, Practical Monitoring Approaches

None of this requires sophisticated infrastructure to start. A few genuinely useful habits, roughly in order of how easy they are to set up:

import pandas as pd

# The original training data, and a batch of recent live requests
train_df = pd.read_csv("student_scores_train.csv")
live_df = pd.read_csv("logged_predictions_last_7_days.csv")

features = ["StudyHours", "Attendance", "PrevScore", "AssignmentsDone"]

# Simple side-by-side comparison of summary statistics
comparison = pd.DataFrame({
    "train_mean": train_df[features].mean(),
    "live_mean": live_df[features].mean(),
    "train_std": train_df[features].std(),
    "live_std": live_df[features].std(),
})
comparison["mean_shift"] = (comparison["live_mean"] - comparison["train_mean"]).abs()

print(comparison.sort_values("mean_shift", ascending=False))

import csv
from datetime import datetime

def log_prediction(features: dict, predicted_score: float):
    """Append one row per prediction — features, output, and a timestamp."""
    row = {**features, "predicted_score": predicted_score, "timestamp": datetime.now().isoformat()}
    with open("prediction_log.csv", "a", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=list(row.keys()))
        if f.tell() == 0:
            writer.writeheader()
        writer.writerow(row)

Calling log_prediction(...) from inside the /predict endpoint (Lesson 22, Section 5) turns every real request into a row that Section 3's comparison script above can analyze later — a genuinely simple addition with real long-term value.

When Is Retraining Actually Needed?

Retraining isn't free — it costs compute time, and every new model version needs to be re-evaluated (Lesson 15's metrics, Lesson 4's cross-validation) before it replaces the one currently in production. A few practical, honest signals for when it's actually worth doing:

The Wider MLOps Landscape

Everything in this lesson can be built with plain pandas, a CSV log file, and a scheduled script — genuinely enough to start. As a project or team grows, dedicated tools exist to handle these same ideas at scale: experiment and model-registry tracking (MLflow), dedicated drift-detection libraries, and general-purpose dashboarding tools for visualizing metrics over time. None of them change the underlying concepts from Sections 2–4 — they automate and scale the same core ideas: compare distributions, log predictions, watch for change.