Support Vector Machines

Find the Widest Street Between Classes

For a binary classification problem, there are infinitely many hyperplanes that can separate two linearly separable classes. SVMs choose the one that maximises the margin — the distance between the hyperplane and the nearest training point from each class.

The intuition: a fatter margin means the decision boundary is further from all training points, giving more room for generalisation. Points that lie exactly on the margin boundaries are called support vectors — they are the only training points that matter for defining the boundary. Delete all other points and you'd get the same hyperplane.

Real data is rarely perfectly separable. SVMs introduce a soft margin via the parameter C:

• High C: penalise misclassifications heavily → narrow margin, fits training data more tightly (risks overfitting)
• Low C: allow some misclassification → wider margin, smoother boundary (risks underfitting)

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

# ALWAYS scale before SVM — it is distance-based
for C in [0.001, 0.1, 1, 10, 100]:
    pipe = Pipeline([
        ('scaler', StandardScaler()),
        ('svm',    SVC(kernel='linear', C=C))
    ])
    score = cross_val_score(pipe, X, y, cv=5).mean()
    print(f"C={C:6}: CV={score:.3f}")

# Number of support vectors used
svc = SVC(kernel='linear', C=1).fit(X_scaled, y)
print(f"Support vectors per class: {svc.n_support_}")

Separate Non-Linear Data Without the Computation Cost

Most real datasets are not linearly separable. The naive fix is to map data into a higher-dimensional space where it becomes separable — for example, adding a feature x² allows a parabola as the decision boundary. But explicitly computing all those new features is expensive.

The kernel trick exploits the fact that the SVM optimisation only ever needs dot products between data points — never the points themselves. A kernel function K(xᵢ, xⱼ) computes what the dot product would be in the high-dimensional space without ever going there. This gives the non-linear power at linear computational cost.

from sklearn.model_selection import GridSearchCV

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('svm',    SVC(kernel='rbf', probability=True))
])

param_grid = {
    'svm__C':     [0.1, 1, 10, 100],
    'svm__gamma': ['scale', 'auto', 0.001, 0.01]
    # 'scale' = 1/(n_features * X.var())  ← usually best default
}

gs = GridSearchCV(pipe, param_grid, cv=5, scoring='accuracy', n_jobs=-1)
gs.fit(X_train, y_train)
print(f"Best params: {gs.best_params_}")
print(f"Best CV score: {gs.best_score_:.3f}")

Classification and Regression with SVMs

sklearn provides SVC (classification) and SVR (regression). SVR introduces an ε-tube around the target: predictions within ε incur no penalty. Only points outside the tube become support vectors.

from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error
import numpy as np

svr_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('svr', SVR(
        kernel='rbf',
        C=10,       # regularisation — same meaning as SVC
        epsilon=0.1 # tube width: predictions within ε are not penalised
    ))
])
svr_pipe.fit(X_train, y_train)
preds = svr_pipe.predict(X_test)
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, preds)):.3f}")

# LinearSVC / LinearSVR: much faster on large datasets
# Uses liblinear solver instead of libsvm — no kernel trick
from sklearn.svm import LinearSVC
fast_svm = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', LinearSVC(C=1, max_iter=2000))
])

When SVMs Are the Right Tool