PCA — Dimensionality Reduction

The Curse of Dimensionality, Briefly

Every distance-based algorithm covered so far — KNN, SVM, K-Means, DBSCAN — relies on the idea that "close together" means "similar." As the number of features grows, that idea quietly breaks down: in high-dimensional space, the distance between the nearest and farthest points from any given point starts to converge, so almost everything ends up looking roughly equally far from everything else. This effect is commonly called the curse of dimensionality.

On top of that, more features usually means more noise, more redundant/correlated columns, slower training, and a higher risk of overfitting relative to the amount of training data available. PCA addresses this directly: instead of discarding features by hand, it mathematically combines all of them into a smaller set of new features that preserve as much of the original information (specifically, variance) as possible.

What PCA Actually Does

PCA looks for the direction in feature space along which the data varies the MOST, and calls that direction the first principal component (PC1). Then it looks for the next-best direction — one that's perpendicular (uncorrelated) to PC1 — and calls that PC2, and so on. Each principal component is a new axis, built as a weighted combination of the ORIGINAL features, ranked by how much of the data's total variance it captures.

sklearn.decomposition.PCA — Fitting and Explained Variance

Exactly like every distance/variance-based algorithm in this course, features need to be scaled BEFORE running PCA — otherwise a large-range feature would dominate "variance" simply because of its units, not because it's genuinely more informative.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print(f"Original shape: {X_scaled.shape}")
print(f"Reduced shape:  {X_pca.shape}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance captured: {pca.explained_variance_ratio_.sum():.3f}")
# Original shape: (500, 8)
# Reduced shape:  (500, 2)
# Explained variance ratio: [0.42 0.23]
# Total variance captured: 0.650

Choosing How Many Components to Keep

There's no single universally-correct number of components — it depends entirely on the goal. The most common approach: plot CUMULATIVE explained variance against the number of components, and pick the smallest number that reaches a target threshold (often 90–95%).

import numpy as np
import matplotlib.pyplot as plt

pca_full = PCA().fit(X_scaled)
cumulative = np.cumsum(pca_full.explained_variance_ratio_)

plt.plot(range(1, len(cumulative) + 1), cumulative, marker="o")
plt.axhline(y=0.95, color="r", linestyle="--", label="95% threshold")
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
plt.legend()
plt.show()

n_components_95 = np.argmax(cumulative >= 0.95) + 1
print(f"Components needed for 95% variance: {n_components_95}")
# Components needed for 95% variance: 6

Scikit-learn also accepts a FLOAT for n_components, letting PCA figure out the exact count itself — a shortcut for the same idea above.

# n_components between 0 and 1 keeps the SMALLEST number of components
# that reaches at least that much explained variance
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X_scaled)

print(f"Reduced from {X_scaled.shape[1]} to {pca.n_components_} dimensions")
# Reduced from 8 to 6 dimensions

PCA for Visualization — Seeing High-Dimensional Data in 2D

Humans can't look at an 8-dimensional scatter plot — but by keeping only the first 2 (or 3) principal components, whatever structure exists in the full feature space often remains visible in a plot a person can actually read.

import matplotlib.pyplot as plt

pca_2d = PCA(n_components=2)
X_pca = pca_2d.fit_transform(X_scaled)

plt.figure(figsize=(8, 6))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap="viridis", alpha=0.6)
plt.xlabel("Principal Component 1")
plt.ylabel("Principal Component 2")
plt.title("PCA — 2D Projection")
plt.colorbar(scatter, label="Class")
plt.show()

PCA as a Preprocessing Step Before Another Model

The second, very different use of PCA: rather than plotting the result, feed the reduced features directly into a classifier or regressor. This can speed up training, reduce overfitting on noisy or redundant features, and — exactly like StandardScaler in Lesson 16 — belongs INSIDE a Pipeline so it gets refit correctly on each cross-validation fold's training data only.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("pca", PCA(n_components=0.95)),
    ("model", SVC(kernel="rbf"))
])
pipeline.fit(X_train, y_train)

print(f"Components kept: {pipeline.named_steps['pca'].n_components_}")
print(f"Test accuracy: {pipeline.score(X_test, y_test):.3f}")

Lesson Summary