Machine Learning Fundamentals / Section 2 — Supervised Learning Algorithms / Lesson 10
📐 Section 2 · Supervised Learning 🔴 Advanced MODULE 10

Gradient Boosting — XGBoost & LightGBM

⏱️ 27 min read
📖 Sequential Ensembles
🧩 4 Quiz Questions
🏗️ 1 Challenge · 3 tasks
Your progress in Section 271%
🎯 What you'll learn: Gradient boosting builds an ensemble of weak models the opposite way a Random Forest does — one at a time, each one specifically trying to fix the previous one's mistakes, rather than many independent trees voting in parallel. You'll see that core idea, use scikit-learn's built-in GradientBoostingClassifier, and then step up to the two libraries that dominate real-world tabular ML: XGBoost and LightGBM.
Section 1

Boosting vs. Bagging — Sequential, Not Parallel

A Random Forest (an algorithm assumed as prior knowledge from earlier in this track) is a bagging ensemble: it trains many decision trees independently and in parallel, each on a random subset of the data, then averages or votes across all of them. Each tree in a Random Forest has no idea what the other trees are doing.

Boosting works completely differently. It builds trees one at a time, in sequence, and each new tree is trained specifically to correct the errors the ensemble has made SO FAR.

1
Start with a weak first guess
Often just the average of the target — a deliberately simple starting point.
2
Measure the errors (residuals)
Compute how far off the current combined model's predictions are from the true values.
3
Train a new small tree on those errors
This next tree's whole job is predicting the RESIDUALS — the mistakes — not the original target directly.
4
Add it to the ensemble, scaled down
The new tree's predictions are added to the running total, shrunk by a learning rate so no single tree dominates.
5
Repeat
Measure the new (smaller) errors, train another tree to fix THOSE, and keep going for many rounds.
✏️
The essay-revision analogy
Picture writing an essay draft, then handing it to an editor who doesn't rewrite the whole thing — they specifically mark up what's WRONG with this draft. You revise based only on their notes. Then a second editor looks at the revised draft and marks up what's STILL wrong, and you revise again based on their notes. After several rounds of "find what's wrong, fix specifically that," the essay improves in a very targeted way — quite different from asking ten independent writers to each write a full essay and averaging them (which is closer to what bagging/Random Forest does).
🌲
Bagging (Random Forest)
Many trees, trained independently and in PARALLEL, on random subsets. Reduces variance by averaging.
🔁
Boosting (Gradient Boosting)
Many trees, trained SEQUENTIALLY, each correcting the previous ensemble's errors. Reduces bias by iteratively focusing on what's still wrong.
Section 2

sklearn.ensemble.GradientBoostingClassifier

scikit-learn's built-in implementation is a solid way to see the algorithm in action, with a familiar .fit()/.predict() interface.

gradient_boosting.py
PYTHON 
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = GradientBoostingClassifier(
    n_estimators=100,     # how many sequential trees to build
    learning_rate=0.1,    # how much each tree's correction counts
    max_depth=3,          # each individual tree stays shallow ("weak learner")
    random_state=42
)
model.fit(X_train, y_train)

print("Test accuracy:", model.score(X_test, y_test))
🔢
n_estimators
How many sequential trees get built. More trees = more opportunities to correct errors, but also more overfitting risk and longer training.
📉
learning_rate
Shrinks each tree's contribution to the running total. Smaller values need more trees to reach the same performance, but often generalize better — a direct tradeoff, similar in spirit to Lesson 6's gradient descent learning rate.
🌱
max_depth
Each individual tree is usually kept SHALLOW on purpose (a "weak learner") — the ensemble's power comes from combining many weak trees sequentially, not from any one tree being powerful alone.
⚠️
Gradient boosting overfits differently than a single deep tree
Because each round chases the previous round's remaining errors, too many rounds (too large n_estimators) can eventually start fitting noise in the training data — the classic overfitting signature from Lesson 5, where training score keeps climbing while test score stalls or drops. Watching the train/test gap as n_estimators grows is exactly the right diagnostic.
Section 3

XGBoost — XGBClassifier

XGBoost ("Extreme Gradient Boosting") is a separate, third-party library — not part of scikit-learn — that implements gradient boosting with major engineering optimizations for speed and performance. It's installed separately (pip install xgboost) but designed to feel familiar if you already know scikit-learn's API.

xgboost_classifier.py
PYTHON 
# pip install xgboost
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = XGBClassifier(
    n_estimators=200,
    learning_rate=0.1,
    max_depth=4,
    subsample=0.8,          # use 80% of rows per tree — adds randomness, reduces overfitting
    colsample_bytree=0.8,  # use 80% of features per tree, same idea
    eval_metric="logloss",
    random_state=42
)
model.fit(X_train, y_train)

print("Test accuracy:", model.score(X_test, y_test))
print("Feature importances:", model.feature_importances_)
✨
XGBoost adds built-in regularization
Beyond the standard gradient boosting parameters, XGBoost's objective function includes explicit L1/L2 regularization terms (conceptually similar to Lesson 5's Ridge/Lasso) baked directly into how each tree is built — one reason it tends to be less overfitting-prone than a naive gradient boosting implementation at comparable settings.
Section 4

LightGBM — LGBMClassifier

LightGBM ("Light Gradient Boosting Machine"), from Microsoft, is another popular third-party gradient boosting library (pip install lightgbm), built around a histogram-based tree-building strategy that makes it especially fast on large datasets.

lightgbm_classifier.py
PYTHON 
# pip install lightgbm
from lightgbm import LGBMClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LGBMClassifier(
    n_estimators=200,
    learning_rate=0.1,
    num_leaves=31,      # LightGBM grows trees leaf-wise, so leaves (not depth) is the main size control
    random_state=42
)
model.fit(X_train, y_train)

print("Test accuracy:", model.score(X_test, y_test))
print("Feature importances:", model.feature_importances_)
📝
num_leaves vs. max_depth
Most tree-based models (including XGBoost's default mode) grow LEVEL-wise — expanding an entire depth layer before going deeper. LightGBM instead grows LEAF-wise by default — always splitting whichever single leaf would reduce error the most, regardless of depth. This can reach lower error with fewer splits, but makes num_leaves (rather than max_depth alone) the more direct control on model complexity and overfitting risk.

Both libraries also support native regression classes — XGBRegressor and LGBMRegressor — with the same constructor style, for continuous targets instead of classification.

Section 5

Why These Libraries Are So Popular

XGBoost and LightGBM are consistently among the most-used tools for structured/tabular data problems, for a handful of concrete, practical reasons — not because of any specific unverified benchmark number:

⚡
Speed
Both are engineered in optimized C++ under the hood, with parallelization and (for LightGBM especially) histogram-based binning that speeds up training on large datasets.
🛡️
Built-in regularization
Explicit controls to fight overfitting are part of the core objective, not an afterthought — helpful straight out of the box.
🕳️
Native missing-value handling
Both can learn how to route missing values during training, often without needing separate imputation (Lesson 3) beforehand.
⚠️
No specific accuracy number is a real claim here
These libraries are widely used in machine learning competitions and industry tabular-data work — that's a real, well-known pattern in the field. But no specific accuracy percentage, leaderboard rank, or "beats X by Y%" claim is being made anywhere in this lesson. Any output shown here is illustrative of the code's mechanics on a toy dataset, not a verified benchmark result.
Section 6

Lesson Summary

✅Boosting trains trees SEQUENTIALLY, each correcting the previous ensemble's errors — unlike Random Forest's parallel bagging.
✅GradientBoostingClassifier from scikit-learn implements the core algorithm with n_estimators, learning_rate, and max_depth.
✅XGBoost (XGBClassifier) and LightGBM (LGBMClassifier) are third-party libraries built for speed, built-in regularization, and native missing-value handling.
✅LightGBM grows trees leaf-wise (num_leaves), while most gradient boosting implementations grow level-wise (max_depth).
🧩 Knowledge Check — Lesson 10
4 questions on boosting, XGBoost, and LightGBM.
1. What is the key difference between boosting and Random Forest's bagging?
2. In GradientBoostingClassifier, what does the learning_rate parameter control?
3. Which import correctly loads XGBoost's classifier?
4. What makes LightGBM's default tree-growing strategy different from most other gradient boosting implementations?
💪
Try It Yourself — Lesson 10
Compare boosting implementations · Advanced Level

These tasks compare the three boosting implementations from this lesson.

Task 1: Three-way comparison 🥊

On the same train/test split, fit GradientBoostingClassifier, XGBClassifier, and LGBMClassifier with roughly matching settings (n_estimators=100, learning_rate=0.1). Compare test accuracy and rough training time for each. Note this as an exploration of YOUR dataset, not a general ranking claim.
Task 2: Sweep n_estimators and watch for overfitting 📈

Using GradientBoostingClassifier, train with n_estimators in [10, 50, 200, 500], recording train and test accuracy each time. At what point (if any) does the train/test gap start widening — the overfitting signature from Lesson 5?
Task 3: Compare num_leaves in LightGBM 🍃

Fit LGBMClassifier with num_leaves at 7, 31, and 127, keeping other settings fixed. How does test accuracy change, and does a very large num_leaves show signs of overfitting?
💡 Show hints if you're stuck
  • Task 1: Results vary by dataset — the point of this task is practicing the parallel API across three libraries, not proving one is universally best.
  • Task 2: A widening gap at very high n_estimators (like 500) is common — more sequential rounds mean more chances to fit noise in the training data.
  • Task 3: num_leaves=127 is quite large for a small dataset and is the value most likely to show a train/test gap; num_leaves=7 is the most conservative/regularized option.
Finished this lesson?
Mark it complete to track your progress.
🎉

Lesson 10 Complete!

You now understand sequential boosting vs. parallel bagging, scikit-learn's GradientBoostingClassifier, and both XGBoost's and LightGBM's real APIs. Next: rather than guessing hyperparameters, you'll learn to search for them systematically.

Module 10 of 24 Section 2 — Supervised Learning Algorithms