Dimensionality Reduction

Why Adding Features Can Hurt

Adding features to a dataset seems obviously helpful — more information, right? But as dimensionality grows, something counterintuitive happens: data becomes sparse. In high dimensions, all points become roughly equidistant from each other. Distance-based algorithms (K-Means, KNN, SVMs) lose their ability to distinguish "nearby" from "far" — because nothing is nearby anymore.

Practically: a dataset with 1,000 samples is dense in 5 dimensions but almost empty in 500 dimensions. You'd need exponentially more data to maintain the same density. Since we can't always collect more data, we reduce dimensions instead.

Rotate Features to Maximise Variance

PCA finds the directions in feature space that explain the most variance. The first principal component (PC1) points in the direction of maximum variance. PC2 points in the direction of maximum remaining variance, orthogonal to PC1. And so on.

Each principal component is a linear combination of the original features. PCA is a rotation of the coordinate system, not a selection of original features. You lose interpretability but gain a compact representation that captures most of the signal.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

# Always scale before PCA — variance is scale-dependent
X_scaled = StandardScaler().fit_transform(X)

# Step 1: fit full PCA to inspect variance
pca_full = PCA()
pca_full.fit(X_scaled)

# Cumulative explained variance — choose where it crosses 0.95
cumvar = np.cumsum(pca_full.explained_variance_ratio_)
n_components_95 = np.argmax(cumvar >= 0.95) + 1
print(f"{n_components_95} components explain 95% of variance")

# Step 2: fit PCA with chosen components
pca = PCA(n_components=n_components_95)
X_pca = pca.fit_transform(X_scaled)
print(f"Reduced: {X_scaled.shape[1]}D → {X_pca.shape[1]}D")

# Shorthand: keep enough for 95% variance automatically
pca_auto = PCA(n_components=0.95)
X_pca2 = pca_auto.fit_transform(X_scaled)

from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca',    PCA(n_components=0.95)),  # fit on train fold only
    ('model',  RandomForestClassifier(n_estimators=100))
])
# PCA fitted on each training fold — no test data leaks into components

Visualise High-Dimensional Clusters in 2D

PCA is linear — it can only capture linear structure. t-SNE and UMAP are non-linear methods designed primarily for visualisation, reducing data to 2D or 3D while preserving local neighbourhood structure (nearby points in high dimensions stay nearby in 2D).

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

# t-SNE: perplexity ≈ expected cluster size (5–50)
tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_2d = tsne.fit_transform(X_scaled)

plt.figure(figsize=(8, 6))
plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y, cmap='tab10', s=15)
plt.title('t-SNE 2D projection')
plt.show()

# UMAP — pip install umap-learn
import umap
reducer = umap.UMAP(n_components=2, random_state=42)
X_umap = reducer.fit_transform(X_scaled)

Reduction Helps Sometimes — and Hurts Other Times

Dimensionality reduction is not a step you should apply by default. Here's the decision:

• Good use cases: features > samples (genomics, NLP bag-of-words), preprocessing for distance-based models (KNN, K-Means, SVM), visualising cluster structure, compressing storage for many-feature datasets
• Bad use cases: you care about which original features matter (PCA removes that), you have few features already, you're using tree-based models (they handle high dimensions natively)

The canonical rule: tree-based models (Random Forest, XGBoost) handle high dimensions well — don't apply PCA before them. Distance-based models (SVM with RBF, KNN, K-Means) benefit significantly from dimensionality reduction.