Model Evaluation & Tuning
Systematic Hyperparameter Optimisation
Default hyperparameters are almost never optimal. GridSearchCV exhaustively tests every combination you specify, using cross-validation to prevent overfitting to the test set. For small grids (< ~100 combinations), it's perfect. For large grids, RandomizedSearchCV samples a fixed number of random combinations — in practice it often finds a nearly-as-good solution in 10% of the compute time.
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV from scipy.stats import loguniform, randint # ── Grid search: small grid, exhaustive ───────────────── param_grid = { 'model__n_estimators': [100, 200, 300], 'model__max_depth': [3, 5, 7, None], 'model__min_samples_split': [2, 5], } grid = GridSearchCV(pipe, param_grid, cv=5, scoring='f1', n_jobs=-1) grid.fit(X_train, y_train) print("Best params:", grid.best_params_) print("Best CV F1:", grid.best_score_.round(4)) # ── Random search: large continuous space ─────────────── param_dist = { 'model__n_estimators': randint(100, 800), 'model__max_depth': randint(2, 12), 'model__learning_rate': loguniform(1e-3, 0.3), 'model__subsample': [0.7, 0.8, 0.9, 1.0], } rnd = RandomizedSearchCV(pipe, param_dist, n_iter=60, cv=5, scoring='roc_auc', n_jobs=-1, random_state=42) rnd.fit(X_train, y_train)
pipe__step__param notation.Diagnosing Bias vs Variance
A learning curve plots training score and validation score as a function of training set size. It tells you definitively whether your model is underfitting (high bias) or overfitting (high variance) — and whether more data would help.
- High bias (underfitting): both curves converge to a poor score — more data won't help, you need a more complex model or better features
- High variance (overfitting): train score is high but val score is low, large gap — regularise, reduce features, or collect more data
- Good fit: both curves converge to a high score, small gap
from sklearn.model_selection import learning_curve import matplotlib.pyplot as plt train_sizes, train_scores, val_scores = learning_curve( model, X_train, y_train, train_sizes=np.linspace(0.1, 1.0, 8), cv=5, scoring='f1', n_jobs=-1 ) train_mean = train_scores.mean(axis=1) val_mean = val_scores.mean(axis=1) plt.plot(train_sizes, train_mean, label='Train') plt.plot(train_sizes, val_mean, label='Validation') plt.xlabel('Training set size'); plt.ylabel('F1 Score') plt.legend(); plt.title('Learning Curve'); plt.show()
When 99% Accuracy Is Useless
Fraud detection, disease diagnosis, churn prediction — in all these datasets, the positive class is rare (1–5%). A model that predicts "not fraud" for every row achieves 99% accuracy but catches zero fraud. You need precision/recall and F1 as metrics, and you need to actively address the imbalance.
Three approaches: class_weight='balanced' tells the model to upweight minority class samples during training (free, zero extra complexity). SMOTE creates synthetic minority samples by interpolating between real examples. Threshold tuning shifts the decision boundary after training to prioritise recall over precision.
from sklearn.ensemble import RandomForestClassifier from imblearn.over_sampling import SMOTE # pip install imbalanced-learn from sklearn.metrics import classification_report, roc_curve # ── Strategy 1: class_weight='balanced' ───────────────── model = RandomForestClassifier(n_estimators=200, class_weight='balanced') model.fit(X_train, y_train) # ── Strategy 2: SMOTE oversampling ────────────────────── sm = SMOTE(random_state=42, k_neighbors=5) X_res, y_res = sm.fit_resample(X_train, y_train) # SMOTE must be applied ONLY to training data, never test data model2 = RandomForestClassifier(n_estimators=200) model2.fit(X_res, y_res) # ── Strategy 3: threshold tuning ──────────────────────── y_proba = model.predict_proba(X_test)[:, 1] fpr, tpr, thresholds = roc_curve(y_test, y_proba) # J-statistic: maximise TPR - FPR (Youden index) best_thresh = thresholds[(tpr - fpr).argmax()] y_pred_tuned = (y_proba >= best_thresh).astype(int) print(classification_report(y_test, y_pred_tuned))
Accuracy Is Not Enough
Picking the scoring metric for GridSearchCV matters as much as the grid itself — you tune towards whatever you optimise. For imbalanced classification, use F1 (harmonic mean of precision and recall) or ROC-AUC. For fraud detection, maximise recall at fixed precision. For ranking, use ROC-AUC.
from sklearn.metrics import ( accuracy_score, # balanced datasets only f1_score, # imbalanced: harmonic mean precision/recall roc_auc_score, # ranking quality regardless of threshold average_precision_score, # PR-AUC: imbalanced + threshold matters classification_report # full per-class breakdown ) # f1_score for multi-class f1_macro = f1_score(y_test, y_pred, average='macro') # unweighted avg f1_weighted = f1_score(y_test, y_pred, average='weighted') # support-weighted # Custom scorer for GridSearchCV from sklearn.metrics import make_scorer recall_scorer = make_scorer(f1_score, pos_label=1) grid = GridSearchCV(pipe, param_grid, cv=5, scoring=recall_scorer)