Decision Trees & Random Forests

How a Decision Tree Splits Data

A decision tree builds itself by repeatedly asking the single best yes/no question it can find about the data — the question that does the most to separate the classes — then repeating that process on each resulting group.

No deep math is required to use a tree — scikit-learn computes every candidate split's impurity internally. The intuition to keep is simple: at every step, the tree asks whatever single question best separates the remaining classes.

DecisionTreeClassifier in Scikit-Learn

This lesson uses load_breast_cancer(), one of scikit-learn's small built-in real-world datasets — 30 numeric measurements from breast mass scans, with a binary target (malignant / benign). It's a genuine, commonly-used teaching dataset bundled directly with the library, not synthetic data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X, y = data.data, data.target
print("Features:", X.shape[1], "| Samples:", X.shape[0])
print("Classes:", data.target_names)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# max_depth caps how many questions deep the tree can go
dt = DecisionTreeClassifier(max_depth=3, random_state=42)
dt.fit(X_train, y_train)

print(f"Train accuracy: {dt.score(X_train, y_train):.3f}")
print(f"Test accuracy:  {dt.score(X_test, y_test):.3f}")

Overfitting Risk in Deep Trees

A tree with no depth limit will keep splitting until every leaf is perfectly pure — often down to leaves containing just one or two training examples. That's a textbook case of overfitting: the tree has essentially memorized the training set's noise, rather than learning a pattern that generalizes.

Random Forest — An Ensemble of Trees

A Random Forest trains many decision trees — often hundreds — and combines their predictions, usually by majority vote for classification. The trick that makes this actually help, rather than just repeating the same overfit tree many times, is called bagging.

from sklearn.ensemble import RandomForestClassifier

# n_estimators = how many trees to train and vote together
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

print(f"Train accuracy: {rf.score(X_train, y_train):.3f}")
print(f"Test accuracy:  {rf.score(X_test, y_test):.3f}")

# Compare directly to the single tree from Section 2 — illustrative,
# not a guaranteed result on every dataset, but forests are frequently more robust
# than a lone unrestricted tree, especially with many features like this one.
print(f"Single tree test accuracy: {dt.score(X_test, y_test):.3f}")

Feature Importance

A useful side benefit of tree-based models: after fitting, they can report how much each feature contributed to reducing impurity across all the splits that used it — a rough but genuinely useful measure of which inputs mattered most.

import numpy as np

importances = rf.feature_importances_
# argsort() gives ascending order; [-5:] takes the top 5; [::-1] reverses to descending
top5 = importances.argsort()[-5:][::-1]

print("Top 5 most important features:")
for i in top5:
    print(f"  {data.feature_names[i]}: {importances[i]:.4f}")