3.3 — SVMs for Classification and Regression

Multi-class with SVC

SVMs are natively binary classifiers. Scikit-learn's SVC extends to multi-class using one-vs-one: train a separate binary classifier for every pair of classes, then combine their votes. For k classes, that's k(k-1)/2 classifiers — fine for a small number of classes, but it grows quickly as classes increase.

SVR — the epsilon-insensitive tube

Instead of fitting a separating boundary, SVR fits a "tube" of width epsilon around the regression line. Points inside the tube incur zero loss; only points outside it contribute to the error, and only their distance beyond the tube matters. This makes SVR robust to small noise around the true relationship, while still penalizing larger deviations.

from sklearn.svm import SVC, SVR

clf = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True)
clf.fit(X_train, y_train)
clf.predict_proba(X_test)   # requires probability=True, adds training cost

reg = SVR(kernel='rbf', C=1.0, epsilon=0.1)
reg.fit(X_train, y_train)

Probability calibration

SVMs don't natively output probabilities — the core algorithm just produces a signed distance from the decision boundary (decision_function). To get predict_proba, scikit-learn fits an auxiliary calibration model (Platt scaling — a logistic regression on the decision function output) on top of that distance.

Takeaways

• One-vs-one extends binary SVMs to multi-class, at a cost that grows quadratically with class count.
• SVR's epsilon tube gives built-in tolerance to small noise, unlike standard squared-error regression.
• SVM predict_proba is a calibrated approximation, not a native output — treat it with appropriate skepticism.