Support Vector Machines (SVM)

The Core Idea — Maximum Margin

For a classification problem with two classes, there are usually MANY possible straight lines (or in higher dimensions, "hyperplanes") that separate the two classes correctly. An SVM asks a sharper question: of all the boundaries that separate the classes, which one leaves the biggest cushion of empty space on either side before hitting a data point?

Why does a wider margin matter in practice? Intuitively, a boundary with more breathing room is less sensitive to small variations or noise in new, unseen data — it's the geometric version of the "generalizes well" idea from Section 1's overfitting lesson.

Support Vectors — Only the Closest Points Matter

Here's the feature that gives the algorithm its name. Once the maximum-margin hyperplane is found, only the data points that sit EXACTLY on the edge of the margin — the closest points from each class — actually determine where that boundary sits. These points are called the support vectors.

Real-world data is rarely perfectly separable. A "soft margin" SVM allows some points to violate the margin, or even land on the wrong side, in exchange for a wider, more generalizable margin overall — that tradeoff is controlled by the C parameter, covered in Section 4.

The Kernel Trick — Handling Data That Isn't Linearly Separable

A straight line (or flat hyperplane) can't separate every dataset — imagine one class forming a ring around the other. The kernel trick is what lets SVMs handle exactly this kind of data, without needing heavy new math to follow it: the intuition is what matters here.

sklearn.svm.SVC in Practice

scikit-learn's SVC (Support Vector Classifier) implements everything above behind a familiar .fit() / .predict() interface. The kernel parameter picks the strategy from Section 3.

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# SVM is distance-based, so ALWAYS scale features first (Lesson 3)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Linear kernel — a straight decision boundary
svm_linear = SVC(kernel="linear", C=1.0)
svm_linear.fit(X_train_scaled, y_train)
print("Linear kernel test accuracy:", svm_linear.score(X_test_scaled, y_test))

# RBF kernel — a flexible, non-linear decision boundary
svm_rbf = SVC(kernel="rbf", C=1.0, gamma="scale")
svm_rbf.fit(X_train_scaled, y_train)
print("RBF kernel test accuracy:", svm_rbf.score(X_test_scaled, y_test))

The illustrative output below shows the SHAPE of a comparison you'd run on a toy dataset — not a benchmark claim about which kernel is universally better:

# Linear kernel test accuracy: 0.87
# RBF kernel test accuracy: 0.91
# -> On THIS toy dataset the RBF kernel captured a curved boundary the
#    linear kernel couldn't — but the linear kernel would win on data
#    that's actually linearly separable, and trains faster besides.

The C Parameter — Regularization for SVMs

C controls the tradeoff between a wide margin and correctly classifying every single training point — the "soft margin" idea from Section 2. It's the SVM equivalent of the regularization strength (alpha) covered for Ridge/Lasso back in Lesson 5.

from sklearn.svm import SVC

# Compare a small vs. large C on the same scaled data
for c_value in [0.01, 1.0, 100]:
    model = SVC(kernel="rbf", C=c_value)
    model.fit(X_train_scaled, y_train)
    train_acc = model.score(X_train_scaled, y_train)
    test_acc = model.score(X_test_scaled, y_test)
    print(f"C={c_value}: train={train_acc:.2f}, test={test_acc:.2f}")

Lesson Summary