Capstone:
Deep Learning Project

This capstone has no single prescribed project — you choose a track that matches your interests and demonstrates mastery of this tier's material. Each track requires you to apply concepts from at least 4 different modules. Pick the one that excites you; motivation matters more than perfectly optimal selection.

• Use a real dataset of meaningful size (not the same toy dataset used throughout the tier — find something new on Kaggle, HuggingFace Datasets, or similar).
• Apply at least one regularization technique from Module 4 and justify the choice based on observed overfitting.
• Use a learning rate schedule from Module 2, and explain why you chose it.
• Train with mixed precision (Module 11) and log the experiment with a tracking tool (Module 11).
• Include a baseline comparison — e.g., a simpler model or a "dumb" baseline (predict the most common class) — to contextualize your model's performance.
• Document failure modes — what didn't work, and your hypothesis for why.

Regardless of track, every ML project follows the same pipeline. This lesson is your structured checklist for executing it well — the same process used in real applied ML work, scaled down to capstone size.

• 1Data exploration and cleaning. Look at your data before modeling anything. Check class balance, missing values, outliers, label quality. Visualize a sample of examples. Split into train/val/test before any preprocessing decisions that could leak information.
• 2Baseline first. Before building your real model, establish a trivial baseline (predict the majority class, or a simple linear model). This tells you what "no learning" looks like and gives a sanity floor for your real model's performance.
• 3Architecture design. Choose an architecture appropriate to your track and data size — don't reach for the largest model you can build. Start smaller than you think you need; scale up only if underfitting.
• 4Overfit a tiny batch first. Before a full training run, verify your model can memorize 10 examples (Lesson 11.3's sanity check). This catches implementation bugs before they waste hours of compute.
• 5Full training with tracking. Train with logged metrics, learning rate schedule, mixed precision, and checkpointing of the best validation score. Expect to iterate — your first full run is rarely your best.
• 6Hyperparameter iteration. Based on the train/val loss curves, diagnose overfitting or underfitting and adjust regularization, model capacity, or learning rate accordingly. Document what you tried and why.
• 7Final evaluation on the held-out test set. Only touch the test set once, at the very end, with your final chosen model. This is your unbiased estimate of real-world performance.

# ── capstone_project.py — skeleton structure ──────────────────────────
import torch
import torch.nn as nn
import wandb

# 1. DATA ────────────────────────────────────────────────────────────
# train_loader, val_loader, test_loader = load_and_split_data(...)
# Inspect class balance, visualize samples, check for leakage

# 2. BASELINE ────────────────────────────────────────────────────────
def majority_class_baseline(val_loader):
    # Compute val accuracy if you always predict the most common class
    pass

# 3. MODEL ───────────────────────────────────────────────────────────
model = YourModel(...).to('cuda')

# 4. SANITY CHECK — overfit a tiny batch ────────────────────────────
# (See Lesson 11.3 code — run this before anything else)

# 5. FULL TRAINING ───────────────────────────────────────────────────
wandb.init(project="capstone", config={...})
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)
early_stop = EarlyStopping(patience=10, path='best_model.pt')   # Module 4

for epoch in range(100):
    train_loss = train_one_epoch(model, train_loader, optimizer)
    val_loss, val_metric = validate(model, val_loader)
    scheduler.step()
    wandb.log({"train_loss": train_loss, "val_loss": val_loss, "val_metric": val_metric})

    if early_stop.step(val_loss, model, epoch):
        break

# 6. FINAL TEST EVALUATION — touch this ONCE ───────────────────────
model.load_state_dict(torch.load('best_model.pt'))
test_loss, test_metric = validate(model, test_loader)
print(f"Final test performance: {test_metric:.4f}")

A model that performs well but that you can't explain is a liability — you can't debug it, trust it, or improve it systematically. This lesson covers two concrete interpretability techniques, one for CNNs and one for transformers, that you'll apply to your capstone model.

Gradient-weighted Class Activation Mapping shows where in an image a CNN focused to make its prediction. It uses the gradient of the predicted class score with respect to the final convolutional layer's feature maps, producing a heatmap overlaid on the original image.

import torch
import torch.nn.functional as F

class GradCAM:
    def __init__(self, model, target_layer):
        self.model = model
        self.gradients = None
        self.activations = None
        target_layer.register_forward_hook(self.save_activation)
        target_layer.register_full_backward_hook(self.save_gradient)

    def save_activation(self, module, inp, out):
        self.activations = out.detach()

    def save_gradient(self, module, grad_in, grad_out):
        self.gradients = grad_out[0].detach()

    def generate(self, input_tensor, class_idx):
        output = self.model(input_tensor)
        self.model.zero_grad()
        output[0, class_idx].backward()   # gradient of class score

        # Global average pool the gradients → importance weight per channel
        weights = self.gradients.mean(dim=(2, 3), keepdim=True)
        # Weighted combination of activation maps
        cam = (weights * self.activations).sum(dim=1, keepdim=True)
        cam = F.relu(cam)   # only positive influence matters
        cam = F.interpolate(cam, size=input_tensor.shape[-2:], mode='bilinear')
        cam = (cam - cam.min()) / (cam.max() - cam.min() + 1e-8)  # normalize to [0,1]
        return cam.squeeze().cpu().numpy()

# Usage
gradcam = GradCAM(model, target_layer=model.layer3[-1])  # last conv block
heatmap = gradcam.generate(input_image, class_idx=predicted_class)
# Overlay heatmap on original image with matplotlib for visualization

Since attention weights are already probability distributions over input tokens, visualizing them is direct — no special technique needed beyond extracting and plotting the attention matrices you already computed during the forward pass.

import torch
import matplotlib.pyplot as plt

# Assuming your MultiHeadAttention (Module 7) returns attention weights
model.eval()
with torch.no_grad():
    output, attn_weights = model.forward_with_attention(input_tokens)
    # attn_weights: (batch, n_heads, seq_len, seq_len)

# Visualize one head's attention pattern as a heatmap
head_idx = 0
attn_matrix = attn_weights[0, head_idx].cpu().numpy()  # (seq_len, seq_len)

fig, ax = plt.subplots(figsize=(8, 8))
im = ax.imshow(attn_matrix, cmap='viridis')
ax.set_xticks(range(len(tokens)))
ax.set_yticks(range(len(tokens)))
ax.set_xticklabels(tokens, rotation=90)
ax.set_yticklabels(tokens)
ax.set_xlabel("Key (attended to)")
ax.set_ylabel("Query (attending from)")
plt.colorbar(im)
plt.title(f"Attention head {head_idx}")
plt.savefig("attention_heatmap.png")

A project that isn't communicated clearly might as well not exist, for the purposes of a portfolio or sharing your work. This final lesson covers how to write up and present your capstone in a way that demonstrates not just that the model works, but that you understand why it works the way it does.

• 1Problem statement. What are you predicting/generating, and why does it matter? 2-3 sentences, no jargon.
• 2Data. Source, size, key statistics, any cleaning/preprocessing decisions and why. Include a sample visualization.
• 3Approach. Architecture choice and justification — why this architecture for this problem? Key hyperparameters and how you chose them (grid search, intuition from the tier's content, etc).
• 4Results. Final metrics vs your baseline. Training/validation curves. Comparison to any alternative approaches you tried.
• 5Interpretation. Your Grad-CAM/attention analysis from Lesson 12.3. What did the model learn? Where does it fail, and what's your hypothesis why?
• 6Limitations and future work. Honest accounting of what you'd do differently with more time/compute/data. This signals maturity, not weakness.