Anomaly Detection

What Anomaly Detection Is Used For

An anomaly (or outlier) is a data point that deviates so much from the rest of the dataset that it looks like it was generated by a different process entirely. Finding them automatically matters across a wide range of practical problems — always framed here generically, without claiming any specific real-world benchmark or company result.

What unites all three: the "normal" cases vastly outnumber the anomalies, the anomalies are the whole point of building the system, and — as Section 5 covers in depth — there's rarely a clean, complete set of labeled anomaly examples to train a normal classifier on.

Statistical Approaches — Z-Score and IQR

The simplest anomaly detectors don't need scikit-learn at all — they're direct applications of the descriptive statistics an introductory stats or data-science course already covers: the mean, standard deviation, and quartiles of a single column.

The Z-Score Method

import numpy as np

z_scores = (df["amount"] - df["amount"].mean()) / df["amount"].std()
outliers = df[np.abs(z_scores) > 3]

print(f"Outliers found via z-score: {len(outliers)}")
# Outliers found via z-score: 7

The IQR (Interquartile Range) Method

Q1 = df["amount"].quantile(0.25)
Q3 = df["amount"].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers = df[(df["amount"] < lower) | (df["amount"] > upper)]
print(f"Outliers found via IQR: {len(outliers)}")
# Outliers found via IQR: 11

sklearn.ensemble.IsolationForest

Z-score and IQR only work one column at a time. Real anomalies are often only visible across MULTIPLE features at once — a transaction amount that's fine on its own, at a time of day that's fine on its own, but the COMBINATION is unusual. IsolationForest handles multiple features natively, and works on a genuinely different principle than every other model in this course.

from sklearn.ensemble import IsolationForest

iso = IsolationForest(n_estimators=100, contamination=0.05, random_state=42)
iso.fit(X)

preds = iso.predict(X)              # 1 = normal, -1 = anomaly
scores = iso.decision_function(X)  # higher = more normal

n_anomalies = (preds == -1).sum()
print(f"Anomalies flagged: {n_anomalies} of {len(X)}")
# Anomalies flagged: 25 of 500

sklearn.svm.OneClassSVM

OneClassSVM takes yet another approach — it adapts the SVM idea from Lesson 16 to a setting with only ONE class ("normal") and no clean negative examples to draw a boundary against. Instead, it tries to learn a boundary that wraps tightly around the region where normal data lives, treating anything outside that boundary as anomalous.

from sklearn.svm import OneClassSVM
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)

oc_svm = OneClassSVM(kernel="rbf", nu=0.05, gamma="scale")
oc_svm.fit(X_scaled)

preds = oc_svm.predict(X_scaled)  # 1 = normal, -1 = anomaly
print(f"Anomalies flagged: {(preds == -1).sum()} of {len(X_scaled)}")

The Practical Challenge — Little or No Labeled Anomaly Data

Every classifier in Section 2 of this course was trained and evaluated with y_test — real labels to check predictions against, and a precision/recall/F1 report (Lesson 13) to summarize results. Anomaly detection usually doesn't have that luxury.

This is exactly why the models in Sections 3 and 4 are UNSUPERVISED (or "semi-supervised" at best) — they never require a full set of labeled anomalies to train on, only an assumption about roughly what fraction of the data is anomalous (contamination / nu). In practice, teams often combine several signals: an unsupervised score from IsolationForest or OneClassSVM, simple statistical rules (Section 2) as a sanity check, and — whenever even a SMALL set of confirmed labels exists — spot-checking the model's top-ranked anomalies against them.

Lesson Summary