Regularization
& Generalization
A network that memorizes training data is useless. This module covers the techniques that make neural networks generalize — learning patterns that transfer to new data, not quirks of the training set.
Dropout
Dropout, introduced by Srivastava et al. in 2014, is one of the most effective and conceptually elegant regularization techniques in deep learning. During training, it randomly sets a fraction of neurons to zero at each forward pass. The network never knows which neurons it can rely on — so it's forced to learn redundant representations that generalize better.
Inference: output = a (no mask, but weights learned to account for dropout)
Inverted dropout (PyTorch default):
Training: ã = (mask / p) ⊙ a ← scale up during training
Inference: output = a ← no scaling needed
Where to apply dropout
After fully-connected layers, not usually after convolutions (which have their own redundancy). Typical values: p=0.5 for FC layers, p=0.1–0.3 for smaller networks or convolutional stages.
Dropout rate tuning
Higher dropout = stronger regularization but slower convergence and potentially worse final accuracy if too high. If validation loss is still much higher than training loss, increase p. If training loss is too high, decrease p.
Spatial dropout
For CNNs: drop entire feature maps rather than individual activations. This prevents adjacent pixels from co-adapting — a more appropriate form of dropout for convolutional features.
Transformers: attention dropout
In transformers, dropout is applied to attention weights and the output of each sublayer. The GPT-2 paper used p=0.1. Modern large models often use little to no dropout, relying on weight decay instead.
import torch
import torch.nn as nn
dropout = nn.Dropout(p=0.5) # 50% of neurons dropped during training
x = torch.ones(10) # all ones for easy visibility
# Training mode: ~50% zeroed, rest scaled up by 1/(1-p) = 2
dropout.train()
print("Train:", dropout(x))
# tensor([2., 0., 2., 0., 2., 2., 0., 2., 0., 2.]) — random each call
# Eval mode: pass-through, no masking
dropout.eval()
print("Eval: ", dropout(x))
# tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]) — unchanged
# In a model
class RegularizedMLP(nn.Module):
def __init__(self):
super().__init__()
self.net = nn.Sequential(
nn.Linear(784, 512),
nn.ReLU(),
nn.Dropout(0.5), # regularize first hidden layer
nn.Linear(512, 256),
nn.ReLU(),
nn.Dropout(0.3), # lighter regularization on second
nn.Linear(256, 10)
# no dropout before the output — you want full capacity here
)
def forward(self, x):
return self.net(x)Batch Normalization
Batch Normalization (Ioffe & Szegedy, 2015) is one of the most impactful innovations in deep learning training. It normalizes the activations within a mini-batch, stabilizing training and enabling much higher learning rates. It's not purely a regularization technique — it fundamentally changes the optimization landscape — but it reduces the need for dropout in many architectures.
μ_B = (1/m) Σ xᵢ ← batch mean
σ²_B = (1/m) Σ (xᵢ − μ_B)² ← batch variance
x̂ᵢ = (xᵢ − μ_B) / √(σ²_B + ε) ← normalize
yᵢ = γ x̂ᵢ + β ← scale and shift (learned)
During training
Uses statistics from the current mini-batch. Also maintains a running mean and variance (exponential moving average) that will be used at inference time.
During inference
Uses the running mean and variance accumulated during training — not the current batch's stats. This is why model.eval() matters: it switches BatchNorm to this inference mode.
Where to place it
Usually after the linear/convolutional layer, before the activation: Linear → BatchNorm → ReLU. Some architectures place it after the activation. Research is inconclusive; pre-activation is now more common.
LayerNorm vs BatchNorm
Transformers use LayerNorm (normalize across features for each sample independently), which works with variable-length sequences and single-sample inference. BatchNorm normalizes across the batch — problematic for sequences and small batches.
import torch
import torch.nn as nn
# For fully-connected layers: nn.BatchNorm1d(num_features)
# For convolutional layers: nn.BatchNorm2d(num_channels)
class BNModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(784, 256)
self.bn1 = nn.BatchNorm1d(256) # normalizes 256 features
self.fc2 = nn.Linear(256, 128)
self.bn2 = nn.BatchNorm1d(128)
self.fc3 = nn.Linear(128, 10)
def forward(self, x):
x = torch.nn.functional.relu(self.bn1(self.fc1(x))) # Linear→BN→ReLU
x = torch.nn.functional.relu(self.bn2(self.fc2(x)))
return self.fc3(x)
# Inspecting running statistics after training
model = BNModel()
model.train()
for _ in range(10):
x = torch.randn(32, 784)
model(x) # running stats accumulate
print("Running mean (first 5):", model.bn1.running_mean[:5])
print("Running var (first 5):", model.bn1.running_var[:5])
print("Learned gamma:", model.bn1.weight[:5]) # γ (scale)
print("Learned beta: ", model.bn1.bias[:5]) # β (shift)L1 / L2 Regularization in a Deep Learning Context
You've seen L1 and L2 regularization in the context of linear models in Tier 2. In deep learning, they appear in a slightly different form and serve a different but related purpose: preventing weights from growing too large, which tends to cause overfitting.
L2 regularization (weight decay) adds a penalty term to the loss: the sum of squared weights scaled by a factor λ. During gradient descent, this adds a term to the weight update that pushes weights toward zero at every step — independently of the loss signal.
L_total = L_task + λ · Σ |wᵢ| (L1 / sparsity)
Gradient update with L2:
w ← w − η · (∂L_task/∂w + 2λw) = w(1 − 2ηλ) − η · ∂L_task/∂w
| Method | Effect on weights | Sparsity | PyTorch |
|---|---|---|---|
| L2 (weight decay) | Pushes all weights toward zero, smoothly | No — small weights, not zero | optimizer weight_decay param |
| L1 | Pushes weights toward zero, creates exact zeros | Yes — feature selection effect | Manual penalty term |
| AdamW | Decouples weight decay from gradient scaling | No | optim.AdamW(..., weight_decay=0.01) |
weight_decay parameter implements L2 regularization — but Adam's adaptive learning rates interact with the penalty, making it less effective. AdamW (Decoupled Weight Decay) separates the decay from the gradient update, making it the correct way to do weight decay with Adam. For transformers and modern models, always use AdamW over Adam.import torch
import torch.nn as nn
import torch.optim as optim
model = nn.Linear(100, 10)
# Correct: AdamW with weight decay
optimizer = optim.AdamW(
model.parameters(),
lr=1e-3,
weight_decay=0.01 # λ = 0.01 is a common starting point
)
# If you want L1 regularization, add it manually to the loss
def l1_penalty(model, lam=1e-4):
l1 = 0
for param in model.parameters():
l1 += param.abs().sum()
return lam * l1
# In training loop:
# loss = criterion(output, targets) + l1_penalty(model)
# Exclude biases and norm layers from weight decay (common practice)
decay_params = [p for n, p in model.named_parameters() if 'bias' not in n]
no_decay_params = [p for n, p in model.named_parameters() if 'bias' in n]
optimizer = optim.AdamW([
{'params': decay_params, 'weight_decay': 0.01},
{'params': no_decay_params, 'weight_decay': 0.0}
], lr=1e-3)Early Stopping and Checkpointing
The simplest regularization technique is also one of the most effective: stop training when you stop improving on the validation set. This is early stopping — and it's free, requiring no change to model architecture, loss function, or optimizer.
The overfitting trajectory
Training loss always decreases. Validation loss decreases, then at some point starts to increase — the model has memorized training data specifics. Early stopping saves the model at the minimum validation loss point.
Patience
Don't stop at the first sign of non-improvement. Use a patience parameter — wait N epochs after the best validation loss before stopping. Common values: 5–20 epochs depending on training noise.
Checkpointing
Save the model whenever validation loss improves. At the end of training (or when early stopping fires), load the best checkpoint — not the last epoch's weights, which are typically overtrained.
What metric to monitor
Usually validation loss. Alternatively validation accuracy for classification, or task-specific metrics (F1, BLEU, etc.). Be consistent — don't switch the metric mid-run.
import torch
class EarlyStopping:
def __init__(self, patience=10, min_delta=1e-4, path='best_model.pt'):
self.patience = patience
self.min_delta = min_delta
self.path = path
self.best_loss = float('inf')
self.counter = 0
self.best_epoch = 0
def step(self, val_loss, model, epoch):
if val_loss < self.best_loss - self.min_delta:
self.best_loss = val_loss
self.best_epoch = epoch
self.counter = 0
torch.save(model.state_dict(), self.path) # save best
print(f" ✓ Saved best model (loss={val_loss:.4f})")
return False # don't stop
else:
self.counter += 1
print(f" No improvement ({self.counter}/{self.patience})")
if self.counter >= self.patience:
print(f"Early stopping. Best epoch: {self.best_epoch}")
return True # stop training
return False
# Usage in training loop
early_stop = EarlyStopping(patience=10, path='best_model.pt')
for epoch in range(200):
# ... training code ...
val_loss = 0.5 # placeholder — compute your real val_loss here
if early_stop.step(val_loss, model, epoch):
break
# After training: load the best checkpoint
model.load_state_dict(torch.load('best_model.pt'))Data Augmentation
The best regularizer is more data. When you can't get more real data, augmentation synthetically creates variation — transforming existing examples in ways that don't change their label but force the model to learn invariances rather than memorizing exact pixel patterns.
| Domain | Common augmentations | Avoid |
|---|---|---|
| Image classification | Random crop, horizontal flip, color jitter, rotation, cutout, mixup | Vertical flip for natural images (rarely valid), extreme color distortion |
| Medical imaging | Rotation, elastic deformation, intensity scaling | Color jitter if color carries diagnostic meaning |
| NLP | Back-translation, synonym replacement, token masking (BERT-style) | Random word insertion that changes sentiment or meaning |
| Audio | Time stretch, pitch shift, noise injection, SpecAugment (frequency masking) | Extreme time stretching that makes speech unintelligible |
from torchvision import transforms
from torchvision.datasets import CIFAR10
from torch.utils.data import DataLoader
# Training transforms — aggressive augmentation
train_transform = transforms.Compose([
transforms.RandomCrop(32, padding=4), # random crop with padding
transforms.RandomHorizontalFlip(p=0.5), # 50% chance of flip
transforms.ColorJitter(
brightness=0.2, contrast=0.2,
saturation=0.2, hue=0.1
), # random color distortion
transforms.RandomRotation(10), # ±10° rotation
transforms.ToTensor(),
transforms.Normalize(
mean=[0.4914, 0.4822, 0.4465], # CIFAR-10 stats
std=[0.2470, 0.2435, 0.2616]
)
])
# Validation transforms — deterministic, no augmentation
val_transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(
mean=[0.4914, 0.4822, 0.4465],
std=[0.2470, 0.2435, 0.2616]
)
])
train_dataset = CIFAR10(root='./data', train=True, transform=train_transform, download=True)
val_dataset = CIFAR10(root='./data', train=False, transform=val_transform)
# MixUp — blend two examples and their labels
import numpy as np
def mixup_data(x, y, alpha=0.4):
lam = np.random.beta(alpha, alpha)
idx = torch.randperm(len(x))
mixed_x = lam * x + (1 - lam) * x[idx]
y_a, y_b = y, y[idx]
return mixed_x, y_a, y_b, lam
# Loss = lam * CE(pred, y_a) + (1-lam) * CE(pred, y_b)