📐 Section 2 · Supervised Learning 🟢 Beginner-Friendly MODULE 08

K-Nearest Neighbors (KNN)

⏱️ 20 min read

📖 Instance-Based Learning

🧩 4 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 243%

🎯 What you'll learn: K-Nearest Neighbors is probably the most intuitive algorithm in this entire course — there's barely any "training" involved at all. You'll learn exactly how it classifies a new point by looking at its neighbors, how to pick K, why feature scaling (Lesson 3) matters more here than almost anywhere else, and both KNeighborsClassifier and its regression sibling KNeighborsRegressor.

Section 1

How KNN Works

KNN's idea can be stated in one sentence: to classify a new point, look at its K closest points in the training data, and go with whichever class is the majority among them.

Pick a K

Choose how many neighbors to consult — e.g. K=5.

Measure distance to every training point

For a new point, compute its distance (usually Euclidean) to every single point already in the training data.

Find the K closest

Sort by distance and take the K nearest training points.

Majority vote

Whichever class appears most often among those K neighbors becomes the prediction for the new point.

🏘️

The "judge a person by their neighbors" analogy

Imagine trying to guess someone's profession just by looking at the 5 people who live closest to them. If 4 out of those 5 neighbors are software engineers, you'd probably guess this person is one too — not because of anything about them directly, but because of the company they keep. KNN literally makes predictions this way: it doesn't learn any general rule about what separates the classes, it just asks "who's nearby, and what are THEY?"

📝

KNN has (almost) no "training" phase

Calling .fit() on a KNN model basically just stores the training data — that's it. All the real work (measuring distances, finding neighbors, voting) happens at PREDICTION time, for every single new point. This is why KNN is called a "lazy learner" or "instance-based" method, in contrast to linear regression or SVM, which do real optimization work upfront during .fit() and predict almost instantly afterward.

Section 2

Choosing K

K is a hyperparameter — you set it, the model doesn't learn it. It has a direct, intuitive relationship to the bias-variance tradeoff from Lesson 5.

🔎

Small K (e.g. K=1)

The prediction is based on just the single nearest point — very sensitive to noise and outliers. High variance, overfitting risk: the decision boundary can look jagged and overly specific to individual training points.

🌐

Large K (e.g. K=n/2)

The prediction averages over a huge neighborhood, smoothing out real local patterns along with the noise. High bias, underfitting risk: the boundary becomes overly smooth and loses detail, and with a K too close to the dataset size, it can start ignoring locality almost entirely.

📈 Illustrative pattern — accuracy vs. K

Picture the x-axis as K increasing from 1 upward, and the y-axis as test accuracy. Accuracy often starts lower at K=1 (noisy, overfit), rises as K grows and averages out noise, peaks somewhere in a moderate range, then starts falling again as K gets so large the model becomes too smoothed-out (underfit). Finding that peak — not just picking the biggest or smallest K — is the actual goal.

✨

Use cross-validation to choose K, not a single split

Exactly like alpha for Ridge/Lasso and C for SVMs, K is best chosen by trying several values and comparing their cross-validated scores (Lesson 4) — Lesson 11's GridSearchCV is built for precisely this kind of sweep.

A practical habit: try an odd K for binary classification, to avoid tie votes splitting 50/50 between the two classes.

Section 3

Why Feature Scaling Matters Enormously for KNN

KNN's entire prediction depends on measuring DISTANCE between points — which means it's extremely sensitive to the scale of each feature. This ties directly back to Lesson 3's preprocessing lesson.

⚠️

An unscaled feature can silently dominate every distance calculation

Imagine predicting loan default risk using income (ranging 20,000–200,000) and credit_score (ranging 300–850). Without scaling, differences in income of a few thousand dollars will swamp the Euclidean distance calculation compared to differences in credit score of even 100+ points — credit_score effectively gets ignored, not because it's less predictive, but purely because of its smaller numeric range.

knn_scaling.py

PYTHON

from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# ALWAYS scale before KNN — fit the scaler on train only, then transform both
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)

print("Scaled KNN test accuracy:", knn.score(X_test_scaled, y_test))

📝

Fit the scaler on training data only

Notice scaler.fit_transform(X_train) but scaler.transform(X_test) — not fit_transform again. The scaler learns its mean/std from the TRAINING data only, then applies that exact same transformation to the test data. Fitting it on the test set too would leak information from the test set into preprocessing, undermining the whole point of holding it out (Lesson 4).

The illustrative comparison below shows the SHAPE of what scaling typically does to KNN accuracy on a dataset with very different feature ranges — not a guaranteed number:

output (illustrative)

OUTPUT

# Unscaled KNN test accuracy: 0.71
# Scaled KNN test accuracy:   0.89
# -> A large accuracy gap like this is a strong sign the unscaled features
#    had very different numeric ranges, distorting the distance calculation.

Section 4

KNeighborsClassifier and KNeighborsRegressor

Everything so far described classification — majority vote among neighbors. KNN generalizes naturally to regression too: instead of voting on a class, average the target values of the K nearest neighbors.

knn_classifier.py

PYTHON

from sklearn.neighbors import KNeighborsClassifier

clf = KNeighborsClassifier(n_neighbors=5)
clf.fit(X_train_scaled, y_train)
predictions = clf.predict(X_test_scaled)
print("Predicted classes:", predictions[:5])

knn_regressor.py

PYTHON

from sklearn.neighbors import KNeighborsRegressor

# Same nearest-neighbor logic, but predicts the AVERAGE target value
# of the K nearest neighbors, instead of a majority-vote class
reg = KNeighborsRegressor(n_neighbors=5)
reg.fit(X_train_scaled, y_train)
predicted_values = reg.predict(X_test_scaled)
print("Predicted values:", predicted_values[:5])

🏠

Housing price

Estimate a house's value from the K most similar nearby sales.

🎬

Recommendations

"Users similar to you liked..." is close to KNN's core intuition.

🩺

Small medical datasets

Simple, interpretable baseline when the dataset is small enough for distance search to be fast.

⚠️

KNN can get slow on large datasets

Because prediction requires comparing a new point against (in the naive case) every training point, KNN's prediction cost grows with the size of the training set — unlike a fitted linear model or SVM, which predict from a fixed, compact set of parameters regardless of how much training data was used. This is one practical reason it's often reached for on smaller or moderate-sized datasets.

Section 5

Lesson Summary

✅KNN classifies a new point by majority vote among its K nearest training points (or averages for regression).

✅Small K risks overfitting (noisy, jagged boundaries); large K risks underfitting (over-smoothed boundaries).

✅Feature scaling is essential — KNN is entirely distance-based, so unscaled features distort the result badly.

✅KNeighborsClassifier for classification, KNeighborsRegressor for regression — same nearest-neighbor logic underneath.

🧩 Knowledge Check — Lesson 8

4 questions on KNN's mechanics, K, and scaling.

1. How does KNN classify a new data point?

2. What risk is most associated with a very SMALL K (like K=1)?

3. Why does feature scaling matter so much for KNN specifically?

4. What does KNeighborsRegressor predict for a new point?

💪

Try It Yourself — Lesson 8

Sweep K and test the scaling effect · Beginner-Friendly

These tasks build direct intuition for K and scaling.

Task 1: Sweep K and find the peak 📈

Using scaled data, train a KNeighborsClassifier for K in [1, 3, 5, 9, 15, 25] and print the test accuracy for each. Which K gives the best test accuracy? Does accuracy fall off again at the largest K, matching the pattern in Section 2?

Task 2: Measure the scaling gap 📏

On a dataset with features of very different numeric ranges, train one KNeighborsClassifier(n_neighbors=5) on unscaled data and another on StandardScaler-scaled data. Report both test accuracies and the size of the gap.

Task 3: Try KNeighborsRegressor 📉

On a regression dataset, fit a KNeighborsRegressor(n_neighbors=5) and compute its test mean_squared_error (Lesson 6). Compare it to a plain LinearRegression fit on the same split — which does better on this dataset?

💡 Show hints if you're stuck

Task 1: Very small K (1, 3) often shows a lower or noisier accuracy than a moderate K (5–15); very large K (25+) often starts declining again as it oversmooths.
Task 2: A meaningful gap (several accuracy points or more) is common whenever the unscaled features have very different ranges — the scaled version should generally do at least as well, often noticeably better.
Task 3: Neither is universally better — it depends on whether the true relationship is closer to linear (favors LinearRegression) or has local, non-linear structure (can favor KNeighborsRegressor).

Finished this lesson?

Mark it complete to track your progress.

Module 08 of 24 Section 2 — Supervised Learning Algorithms

K-Nearest Neighbors (KNN)

How KNN Works

Choosing K

Why Feature Scaling Matters Enormously for KNN

KNeighborsClassifier and KNeighborsRegressor

Lesson Summary

Lesson 8 Complete!