Classification Metrics — Deep Dive

Why Accuracy Alone Can Mislead You

Accuracy — (correct predictions) / (total predictions) — is easy to compute and easy to understand, which is why it's been this course's default metric so far. But accuracy has a blind spot: it treats every mistake the same, and it says nothing about WHICH kind of mistake a model tends to make.

Consider a classifier that predicts whether an email is spam. Two very different models could both score "90% accuracy" — one that correctly catches most spam while rarely bothering real email, and another that is great at recognizing real email but quietly lets more spam through. Accuracy alone can't tell those two models apart. The metrics in this lesson can.

The Confusion Matrix — TP, FP, TN, FN

Every other metric in this lesson is built from four counts. For a binary classifier (predicting a "positive" class, like "spam" or "has disease", against a "negative" class):

scikit-learn computes this directly from true and predicted labels with confusion_matrix:

from sklearn.metrics import confusion_matrix

# y_test and y_pred are 0/1 arrays: 1 = positive class (e.g. "spam")
cm = confusion_matrix(y_test, y_pred)
print(cm)

# [[85  5]
#  [10 50]]
# Illustrative output for a toy dataset. Rows = actual class (0, then 1),
# columns = predicted class (0, then 1), sorted ascending by default.
# -> TN=85, FP=5, FN=10, TP=50

Precision, Recall, and F1-Score

These three metrics each combine TP/FP/TN/FN in a different way, answering three different plain-language questions.

Precision — "Of everything I flagged as positive, how much was right?"

Recall — "Of everything that actually was positive, how much did I catch?"

F1-Score — a single number balancing both

Using the illustrative confusion matrix from Section 2 (TN=85, FP=5, FN=10, TP=50):

from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score

print("Accuracy:", accuracy_score(y_test, y_pred))
print("Precision:", precision_score(y_test, y_pred))
print("Recall:", recall_score(y_test, y_pred))
print("F1-score:", f1_score(y_test, y_pred))

# Accuracy:  0.900   -> (85+50) / (85+5+10+50)
# Precision: 0.909   -> 50 / (50+5)
# Recall:    0.833   -> 50 / (50+10)
# F1-score:  0.870   -> 2 * (0.909*0.833) / (0.909+0.833)

classification_report — Every Metric at Once

Instead of calling four separate functions, classification_report prints precision, recall, F1, and support (the number of true instances) for every class in one readable block — plus accuracy and two kinds of averages.

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred, target_names=["Not Spam", "Spam"]))

ROC Curve and AUC

Precision, recall, and F1 all depend on one fixed decision threshold — usually "predict positive if predicted probability > 0.5". The ROC curve (Receiver Operating Characteristic) instead shows how a classifier performs across EVERY possible threshold at once.

The ROC curve traces (FPR, TPR) pairs as the decision threshold sweeps from 0 to 1. roc_curve computes the raw points; roc_auc_score computes the Area Under that Curve as one summary number.

from sklearn.metrics import roc_curve, roc_auc_score

# predict_proba returns [P(class 0), P(class 1)] per row — take column 1
y_probs = model.predict_proba(X_test)[:, 1]

fpr, tpr, thresholds = roc_curve(y_test, y_probs)
auc = roc_auc_score(y_test, y_probs)

print(f"AUC: {auc:.3f}")
# AUC: 0.923   (illustrative — one toy-dataset run, not a benchmark claim)

Precision vs. Recall — Which Should You Prioritize?

There's no universally "correct" metric to optimize — the right choice depends entirely on which TYPE of mistake, false positives or false negatives, is more costly for the specific problem. Two classic contrasting examples:

Reasoning through this tradeoff — rather than defaulting to accuracy — is the single biggest mindset shift this lesson aims to build. The right metric always follows from the real-world cost of each type of mistake, never from habit.

Lesson Summary