Clustering Algorithms — K-Means, DBSCAN & Hierarchical

From Predicting to Grouping

Sections 1–3 were all SUPERVISED learning: every row of training data came with a correct answer attached, and every metric (accuracy, RMSE, R²) worked by comparing a prediction against that known answer. Clustering is different in a fundamental way — there is no y_train at all. The algorithm only sees feature rows and has to decide, on its own, which rows "belong together."

That changes what "success" even means. There's no ground-truth cluster label to check predictions against, so evaluating a clustering result (Section 5 of this lesson) works completely differently from evaluating a classifier or regressor.

K-Means Revisited — Centroids, K-Means++, and the Elbow Method

K-Means partitions data into k clusters, each represented by a centroid — the mean position of every point currently assigned to it. The algorithm alternates between two steps until the centroids stop moving: assign every point to its nearest centroid, then recompute each centroid as the mean of its newly assigned points.

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)

kmeans = KMeans(n_clusters=4, init="k-means++", n_init=10, random_state=42)
labels = kmeans.fit_predict(X_scaled)

print(f"Cluster centers shape: {kmeans.cluster_centers_.shape}")
print(f"Inertia (WCSS): {kmeans.inertia_:.1f}")
# Cluster centers shape: (4, 2)
# Inertia (WCSS): 187.3

import matplotlib.pyplot as plt

inertias = []
k_range = range(1, 11)
for k in k_range:
    km = KMeans(n_clusters=k, init="k-means++", n_init=10, random_state=42)
    km.fit(X_scaled)
    inertias.append(km.inertia_)

plt.plot(k_range, inertias, marker="o")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Inertia (WCSS)")
plt.title("Elbow Method")
plt.show()

Each additional cluster always lowers inertia — with enough clusters, every point could get its own centroid and inertia hits zero. The elbow method looks for the k where adding another cluster stops buying a big inertia improvement — here, the drop from k=3 to k=4 (-49%) is much larger than the drop from k=4 to k=5 (-18%), so k=4 is the illustrative "elbow."

DBSCAN — Density-Based Clustering

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) takes a completely different approach: instead of measuring distance to a centroid, it looks at local point DENSITY. A cluster is a region where points are packed closely together, separated from other such regions by areas of low density.

Every point ends up in one of three categories: a core point has at least min_samples neighbors within eps; a border point is within eps of a core point but doesn't have enough neighbors itself; and a noise point is neither — it's isolated, and DBSCAN labels it -1 rather than forcing it into a cluster.

from sklearn.cluster import DBSCAN

dbscan = DBSCAN(eps=0.5, min_samples=5)
labels = dbscan.fit_predict(X_scaled)

n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Clusters found: {n_clusters}")
print(f"Noise points: {n_noise}")
# Clusters found: 3
# Noise points: 14

Hierarchical / Agglomerative Clustering — Dendrograms

Agglomerative clustering builds a hierarchy from the bottom up: it starts with every point as its own cluster, then repeatedly merges the two closest clusters together, one merge at a time, until only one giant cluster remains. That entire merge history is drawn as a dendrogram — a tree diagram where the height of each merge shows how far apart the two things being joined were.

from scipy.cluster.hierarchy import dendrogram, linkage
import matplotlib.pyplot as plt

Z = linkage(X_scaled, method="ward")

plt.figure(figsize=(10, 5))
dendrogram(Z)
plt.xlabel("Sample index")
plt.ylabel("Distance")
plt.title("Hierarchical Clustering Dendrogram")
plt.show()

Getting actual cluster assignments back — rather than just a picture — is what sklearn.cluster.AgglomerativeClustering is for. It runs the same bottom-up merging internally, and returns labels for a chosen number of clusters directly, without a separate dendrogram-reading step.

from sklearn.cluster import AgglomerativeClustering

agg = AgglomerativeClustering(n_clusters=4, linkage="ward")
labels = agg.fit_predict(X_scaled)

print(f"Cluster sizes: {[list(labels).count(i) for i in set(labels)]}")
# Cluster sizes: [12, 9, 15, 6]

Evaluating Clusters Without Labels — Silhouette Score

Every metric in Section 3 of this course (accuracy, F1, RMSE, R²) needed a true label to compare against. Clustering has none — so a different kind of metric is needed, one that judges cluster quality using only the data and the assigned labels themselves.

from sklearn.metrics import silhouette_score

score = silhouette_score(X_scaled, labels)
print(f"Silhouette score: {score:.3f}")
# Silhouette score: 0.612   -> reasonably well-separated, compact clusters

Choosing the Right Algorithm

All three algorithms cluster the same kind of data, but they make different trade-offs — the right choice depends on what's known about the data going in.

Lesson Summary