Classical
Machine Learning

Information gain, entropy, and Gini impurity — built up from scratch with a worked example. Reading a visualized decision tree. Why trees are "white box" models that humans can actually inspect.

A deep tree memorizes training data. max_depth, min_samples_leaf, and cost-complexity pruning as controls. Visualizing the bias-variance curve as you change depth.

Why averaging many weak, diverse models beats one strong model. Bootstrap sampling and feature subsampling — two sources of diversity. Out-of-bag error as a free validation set.

Boosting as sequential error correction, not averaging. How XGBoost and LightGBM differ from vanilla GBM. Practical tuning: n_estimators, learning_rate, max_depth — the three you must always set. When LightGBM beats XGBoost (and vice versa).

MDI (mean decrease impurity) importance and why it's biased toward high-cardinality features. Permutation importance as a correction. SHAP values for truly reliable explanations. What "importance" actually tells you and what it doesn't.

Why maximum margin generalizes better than any separating hyperplane. Support vectors — the few points that determine everything. Hard vs soft margin (C parameter). Drawing the geometry by hand before touching code.

Why we need kernels: data that isn't linearly separable. The "feature space trick" — computing dot products in high-dimensional space without explicitly being there. RBF kernel, intuition of gamma. Polynomial kernels. When kernel SVMs are impractical (large N).

SVC for multi-class (one-vs-one strategy). SVR — the epsilon-insensitive tube. Probability calibration (SVM doesn't give probabilities natively, and what predict_proba actually does).

A decision guide, not a vague suggestion. SVMs win on: small-to-medium datasets, high-dimensional sparse data (text), when you need a clean decision boundary. Trees win on: tabular data with mixed types, large N, interpretability requirements. Linear models win on: very large datasets, need for speed, regularized regression.

Initialize → assign → update → repeat. The Lloyd's algorithm, visualized at each step. Why initialization matters (K-Means++ vs random). Convergence and why K-Means can get stuck in local minima.

Inertia (WCSS) and the elbow — why the "elbow" is often ambiguous. Silhouette score as a more principled metric. Silhouette plots for diagnosing cluster quality. Combining both to make a defensible choice.

Agglomerative (bottom-up) vs divisive (top-down). Linkage methods: single, complete, average, Ward — and which one to pick. Reading dendrograms. When hierarchical clustering is better than K-Means (small N, unknown K, nested structure).

Why K-Means fails on non-convex shapes. Core points, border points, and noise (outliers). Epsilon and min_samples — what they control geometrically. DBSCAN vs K-Means: the shape test. When DBSCAN is the right tool (anomaly-adjacent clusters, unknown K, irregular shapes).

Full end-to-end: load an e-commerce dataset → clean → scale → cluster → evaluate → name and describe each segment → visualize with PCA → write one business recommendation per segment. Builds the vocabulary for real consulting work.

Why distance metrics break in high dimensions. Sparsity in high-dimensional space. How many training samples you need as dimensions grow. The practical consequences for k-NN, SVMs, and any distance-based algorithm.

Variance as information. Principal components as directions of maximum variance. Eigendecomposition of the covariance matrix — intuition first, then code. Explained variance ratio and the scree plot. Choosing n_components. Reconstructing original features. What PCA destroys (and why that's usually fine).

PCA is linear — t-SNE and UMAP are not. How t-SNE works (probability distributions over pairwise distances, KL divergence minimization). Why t-SNE results can be misleading (perplexity, random seed, cluster sizes don't mean what you think). UMAP: faster, more globally consistent, better for downstream tasks. When to use each.

The cases where PCA helps: multicollinearity, storage, speed. The cases where it hurts: tree-based models (already handle it), when features are interpretable and you need them to stay so. The wrong reason to use PCA (to avoid feature selection work). A decision checklist.

Ratio features, interaction terms, polynomial features. Domain-driven features (e.g., "days since last purchase" from a timestamp, "price per square meter" from two columns). Binning continuous variables and when it helps. The craft of asking "what would a domain expert notice here?"

One-hot encoding — and why it explodes your feature space. Ordinal encoding — and when ordinality is actually meaningful. Target encoding — powerful and dangerous (why it causes leakage if done wrong). Frequency encoding. Hashing for high-cardinality. A decision tree for choosing the right encoding.

Text: bag of words, TF-IDF as a feature. Dates: extracting day-of-week, month, quarter, is_weekend, days_since_event. Why raw timestamps are useless to tree models. Lag features and rolling statistics for sequential data.

Filter methods: correlation, mutual information, chi-squared. Wrapper methods: recursive feature elimination (RFE). Embedded methods: L1 regularization, tree importance thresholding. SelectFromModel in scikit-learn. The right order: feature creation → selection (not selection → creation).

Start with a raw messy dataset (housing, Titanic, or retail — chosen for realism). Baseline model score with raw features. Then: engineer 8–10 new features, select the best, re-run model. Measure the score delta. Write a short analysis of which features moved the needle and why.

The difference between parameters (learned) and hyperparameters (set). Grid search — exhaustive but combinatorially expensive. Random search — why it often finds better solutions in fewer trials (Bergstra & Bengio, intuition without the paper). Running both inside a Pipeline (critical: tuning outside a pipeline leaks data). Reading GridSearchCV results.

Why random search is still "dumb" — it doesn't learn from previous trials. Gaussian process surrogate model. Acquisition functions (expected improvement). Optuna as a practical implementation. When Bayesian optimization earns its overhead (expensive models, large search spaces).

Training score vs validation score as a function of training set size. Reading the four diagnostic patterns: underfit (both low), overfit (gap), just right (converging high), data-limited (high but would improve with more data). What to do for each. This is the most efficient debugging tool in supervised ML.

Why accuracy is the wrong metric for fraud detection (predicting all-negative is 99.9% accurate). SMOTE — synthetic minority oversampling, step by step. Class weights in sklearn (simpler than SMOTE, often good enough). Threshold tuning on the probability output. Precision-recall curves and PR-AUC as the right metrics. Common mistake: oversampling before cross-validation (data leakage).

The iid (independent, identically distributed) assumption — and how time series violates it. Autocorrelation: today's value depends on yesterday's. Stationarity — why most models need it and what it means. ACF and PACF plots for diagnosing structure in a series.

Additive vs multiplicative decomposition. Classical decomposition in statsmodels. STL decomposition as a more robust alternative. Extracting and removing trend before modeling. Detecting seasonality visually and numerically. Why you care: residuals after decomposition are what your model actually learns from.

AR (autoregressive) — predicting from past values. I (integrated) — differencing to achieve stationarity. MA (moving average) — modeling the error terms. Choosing p, d, q — using ACF/PACF and AIC. Fitting and forecasting with statsmodels. SARIMA for seasonal series. Auto-ARIMA (pmdarima) for when you don't want to pick manually.

Why random shuffling destroys time series evaluation (you'd be predicting the past from the future). Walk-forward validation (expanding window). Sliding window cross-validation. TimeSeriesSplit in scikit-learn. The minimum gap between train and test to avoid look-ahead leakage.

Full pipeline: load retail time series → EDA → decompose → make stationary → fit ARIMA → evaluate with MAE/RMSE → fit LightGBM with lag features as comparison → plot both forecasts with confidence intervals → write a one-page business summary of your predictions and uncertainty.

User-based CF: "users like you also liked…" — cosine similarity between user rating vectors. Item-based CF: "users who liked this also liked…" — more stable, scales better. The user-item matrix. Sparse matrix representations. Cold-start problem: what to do when a new user has no history. Memory-based vs model-based CF.

Using item features (genre, tags, description) rather than user behavior. TF-IDF on item descriptions as feature vectors. Cosine similarity between items. Building a user profile from their liked items. Strengths: no cold-start for items, explainable. Weaknesses: filter bubble, ignores collective intelligence.

SVD (singular value decomposition) for latent factor models — decomposing the user-item matrix into user and item embeddings. Surprise library for implementing SVD-based recommenders. Hybrid: content + collaborative (how real systems work). Evaluation: precision@K, recall@K, NDCG. The offline vs online evaluation distinction.

Choose a dataset (MovieLens or a course dataset). Implement both user-based CF and content-based. Combine into a hybrid. Handle cold-start with popularity fallback. Evaluate with precision@K. Wrap the recommendation function in a simple FastAPI endpoint. Write a brief analysis of where each approach fails.

The most important skill that courses skip. "We want to reduce churn" → what is the prediction task? What's the label? What's the positive class? What's the cost of a false positive vs false negative? What data do we have, and when? Worked examples: fraud detection, churn prediction, demand forecasting. A framing template you can reuse on any project.

Structured approach to each stage. EDA checklist (distributions, missing values, correlations, outliers). Feature engineering from domain knowledge. Model selection with cross-validation — no peeking at test set. Evaluation with the right metric for the business problem. Comparing at least 3 algorithms and justifying your final choice.

Load a saved joblib Pipeline. Build a POST /predict endpoint that accepts JSON, runs the pipeline, returns a prediction with probability. Input validation with Pydantic. A simple health check endpoint. Testing with curl and the FastAPI docs page. This is the minimum viable deployment that demonstrates "I can ship this."

The difference between an ML evaluation report and a business recommendation. Translating RMSE into business impact ("this means our forecasts are off by ±$12k per week on average"). Visualizations that work for executives (not ROC curves). Confidence and uncertainty — how to say "I'm not sure" honestly without losing credibility. Template for a 5-slide results deck.