Tier 3 · Deep Learning & Neural Networks

PyTorch
Fundamentals

The framework that runs research and increasingly production. You've built networks from scratch — now learn the tools professionals use, without the mechanics becoming a black box beneath you.

📚 5 Lessons 🔥 PyTorch ⚡ GPU Lab ⏱ ~4 hours

Lesson 3.1

Tensors and Autograd

PyTorch is built on two foundational ideas: tensors (multi-dimensional arrays, like NumPy but GPU-capable) and autograd (automatic differentiation — the engine that computes gradients without you deriving anything by hand). Every other PyTorch feature builds on these two.

🔥

Why PyTorch over TensorFlow? PyTorch uses dynamic computation graphs — the graph is built on the fly as your code runs, not declared ahead of time. This means standard Python control flow (if/else, for loops) works inside models, debugging is just print statements, and the code looks like normal Python. TensorFlow 2.x adopted this style too, but PyTorch's API is widely considered cleaner. It dominates ML research papers.

Pythontensors — the basics

import torch

# Creating tensors
x = torch.tensor([1.0, 2.0, 3.0])
A = torch.zeros(3, 4)          # (3,4) zeros
B = torch.randn(4, 5)          # (4,5) random normal
C = torch.ones(3, 5)           # (3,5) ones

# Tensor operations — same as NumPy
out = A @ B                     # (3,5) matrix multiply
print(out.shape)                 # torch.Size([3, 5])

# Key difference from NumPy: dtype and device
print(x.dtype)                   # torch.float32
print(x.device)                  # cpu

# Converting between NumPy and PyTorch
import numpy as np
arr = np.array([1, 2, 3], dtype=np.float32)
t   = torch.from_numpy(arr)   # shares memory!
arr2 = t.numpy()               # back to NumPy

Autograd: automatic differentiation

Autograd tracks every operation on tensors that have requires_grad=True. When you call .backward() on a scalar loss, PyTorch automatically computes the gradient of that loss with respect to every parameter in the computation graph — the entire backward pass you wrote by hand in Module 1.

Pythonautograd — automatic gradient computation

import torch

# requires_grad=True tells PyTorch to track this tensor
w = torch.tensor(2.0, requires_grad=True)
x = torch.tensor(3.0)   # no grad needed for input data
b = torch.tensor(1.0, requires_grad=True)

# Forward pass — PyTorch builds the computation graph
z = w * x + b          # z = 7.0
L = z ** 2             # L = 49.0  (our "loss")

# Backward pass — computes all gradients automatically
L.backward()

print(f"dL/dw = {w.grad}")   # tensor(42.)  — matches our chain rule ✓
print(f"dL/db = {b.grad}")   # tensor(14.)  — dL/db = 2z = 14

# IMPORTANT: gradients accumulate by default — always zero before next step
w.grad.zero_()
b.grad.zero_()

# The no_grad context: use during inference (saves memory, faster)
with torch.no_grad():
    prediction = w * x + b   # graph not built, gradients not tracked

⚠️

The most common PyTorch mistake Forgetting to zero gradients before the backward pass. PyTorch accumulates gradients — each call to .backward() adds to .grad rather than replacing it. In a training loop, always call optimizer.zero_grad() at the start of each step, or you'll be training on a mix of old and new gradients.

Lesson 3.2

Building Your First nn.Module

nn.Module is PyTorch's base class for all neural network components. Your layers, your full model, and any custom piece you build all subclass it. The pattern is always the same: define parameters in __init__, define the forward computation in forward.

🏗️

nn.Module

Base class for all models. Handles parameter registration, device movement, saving/loading, and training/eval mode switching automatically when you subclass it correctly.

📦

nn.Linear

A fully-connected (dense) layer: y = xW + b. Takes (in_features, out_features). Parameters are registered automatically — no manual weight creation needed.

🔁

nn.Sequential

Stack layers in order. Cleaner than writing forward() for simple feedforward networks. Each layer's output becomes the next layer's input automatically.

📋

model.parameters()

Returns all learnable parameters in the model — what you pass to the optimizer. PyTorch finds them automatically because you registered them via nn.Module subclassing.

Pythonbuilding a neural network with nn.Module

import torch
import torch.nn as nn
import torch.nn.functional as F

# ── Custom module (explicit control over architecture) ──────────────────
class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        # Layers are registered as attributes — PyTorch tracks their params
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.fc3 = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(p=0.3)

    def forward(self, x):
        x = F.relu(self.fc1(x))     # hidden layer 1 + ReLU
        x = self.dropout(x)          # regularization
        x = F.relu(self.fc2(x))     # hidden layer 2 + ReLU
        x = self.fc3(x)               # output (no activation here)
        return x

# Instantiate and inspect
model = MLP(input_dim=784, hidden_dim=256, output_dim=10)
print(model)
# MLP(
#   (fc1): Linear(in_features=784, out_features=256, bias=True)
#   (fc2): Linear(in_features=256, out_features=256, bias=True)
#   (fc3): Linear(in_features=256, out_features=10, bias=True)
#   (dropout): Dropout(p=0.3, inplace=False)
# )

# Count parameters
n_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Trainable parameters: {n_params:,}")  # 269,322

# Forward pass
x = torch.randn(32, 784)    # batch of 32 images (flattened 28×28)
out = model(x)               # calls forward() automatically
print(out.shape)               # torch.Size([32, 10])

# ── Sequential: cleaner for simple feedforward nets ──────────────────
simple_model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(0.3),
    nn.Linear(256, 10)
)

✅

When to use Sequential vs subclassing Sequential is great when every layer feeds directly into the next with no branching. Subclass nn.Module when you have skip connections (ResNets), multiple inputs, custom logic, or architectures where the forward pass isn't a simple chain. Default to subclassing for anything non-trivial — it's more readable and debuggable.

Lesson 3.3

Loss Functions and Optimizers in PyTorch

PyTorch provides all standard loss functions in torch.nn and all common optimizers in torch.optim. Understanding which loss to use for which task — and why — is as important as the model architecture itself.

Task	Loss Function	PyTorch	Notes
Binary classification	Binary Cross-Entropy	`nn.BCEWithLogitsLoss()`	Includes sigmoid — more numerically stable
Multi-class classification	Cross-Entropy	`nn.CrossEntropyLoss()`	Includes softmax — use raw logits as input
Regression	Mean Squared Error	`nn.MSELoss()`	Sensitive to outliers; consider Huber loss
Regression (robust)	Huber / Smooth L1	`nn.HuberLoss()`	Less sensitive to outliers than MSE

Pythonloss functions and optimizers

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(10, 1)   # simple model for demo

# ── Loss functions ───────────────────────────────────────────────────────
mse_loss = nn.MSELoss()
ce_loss  = nn.CrossEntropyLoss()    # for multi-class
bce_loss = nn.BCEWithLogitsLoss()   # for binary

# CrossEntropyLoss: takes raw logits, NOT softmax probabilities
logits = torch.randn(8, 5)    # batch=8, 5 classes
labels = torch.randint(0, 5, (8,))  # integer class indices
loss = ce_loss(logits, labels)
print(f"CE Loss: {loss.item():.4f}")

# ── Optimizers ────────────────────────────────────────────────────────
sgd   = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
adam  = optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999))
adamw = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

# ── Learning rate schedulers ──────────────────────────────────────────
scheduler = optim.lr_scheduler.CosineAnnealingLR(adam, T_max=100)
# Call scheduler.step() at the end of each epoch

# Warmup + cosine (manual, or use timm's scheduler)
# from timm.scheduler import CosineLRScheduler

⚠️

Common confusion: CrossEntropyLoss input format nn.CrossEntropyLoss expects raw logits (before softmax), not probabilities. It applies log_softmax internally for numerical stability. If you apply softmax to your model output first and then pass it to CrossEntropyLoss, you'll get wrong gradients. Similarly, BCEWithLogitsLoss expects logits, not sigmoid outputs.

Lesson 3.4

Training Loops, from Scratch — Understand the Mechanics

High-level libraries like PyTorch Lightning hide the training loop. Don't use them yet. The training loop in raw PyTorch is short — about 10 lines per iteration — and understanding it is essential for debugging, custom training procedures, and any research work.

The canonical PyTorch training loop

1Set model to train mode — model.train() enables dropout and batch norm's training behavior.
2Zero the gradients — optimizer.zero_grad(). Always first in the step, not last.
3Forward pass — output = model(inputs). Builds the computation graph.
4Compute loss — loss = criterion(output, targets).
5Backward pass — loss.backward(). Computes gradients via autograd.
6Clip gradients (optional) — torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) for stability.
7Update parameters — optimizer.step(). Applies the gradient update.
8Update scheduler (optional) — scheduler.step(). Adjusts learning rate.

Pythoncomplete training loop with validation

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# ── Dummy dataset ──────────────────────────────────────────────────────
X_train = torch.randn(1000, 20)
y_train = (X_train[:, 0] > 0).long()   # binary labels
X_val   = torch.randn(200, 20)
y_val   = (X_val[:, 0] > 0).long()

train_loader = DataLoader(TensorDataset(X_train, y_train), batch_size=32, shuffle=True)
val_loader   = DataLoader(TensorDataset(X_val, y_val),   batch_size=64)

# ── Model, loss, optimizer ─────────────────────────────────────────────
model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 2))
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=20)

# ── Training loop ──────────────────────────────────────────────────────
for epoch in range(20):

    # ── Train ─────────────────────────────────────────────────────────
    model.train()
    train_loss, correct = 0.0, 0

    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()               # 1. zero grads
        outputs = model(X_batch)            # 2. forward
        loss = criterion(outputs, y_batch)  # 3. loss
        loss.backward()                     # 4. backward
        optimizer.step()                    # 5. update

        train_loss += loss.item()
        correct += (outputs.argmax(1) == y_batch).sum().item()

    scheduler.step()   # update LR after each epoch

    # ── Validate ──────────────────────────────────────────────────────
    model.eval()
    val_loss, val_correct = 0.0, 0

    with torch.no_grad():   # no gradient computation during eval
        for X_batch, y_batch in val_loader:
            outputs = model(X_batch)
            val_loss += criterion(outputs, y_batch).item()
            val_correct += (outputs.argmax(1) == y_batch).sum().item()

    train_acc = correct / len(X_train)
    val_acc   = val_correct / len(X_val)
    print(f"Epoch {epoch+1:2d}  train_acc={train_acc:.3f}  val_acc={val_acc:.3f}")

✅

model.train() vs model.eval() These switch behavior of Dropout (active during training, identity during eval) and BatchNorm (uses batch stats during training, running stats during eval). Forgetting to call model.eval() before validation is a subtle bug — dropout randomly zeroes activations, making your validation loss artificially high and non-deterministic.

Lesson 3.5

GPU Acceleration Basics

Training on CPU works for small experiments. For anything real — CNNs on ImageNet, transformers on text — a GPU is required. GPUs have thousands of cores optimized for the kind of matrix operations neural networks are made of. A task that takes 4 hours on CPU takes 4 minutes on a modern GPU.

🎮

Why GPUs?

CPUs have ~16 powerful cores for sequential tasks. GPUs have thousands of simple cores designed for parallel computation — exactly what matrix multiplications and convolutions need.

📌

Device placement

Everything — model AND data — must be on the same device. The most common error is having the model on GPU but forgetting to move data batches to GPU too.

🔀

CUDA vs MPS vs CPU

CUDA = NVIDIA GPUs (most common in ML). MPS = Apple Silicon (M1/M2/M3). Both are accessed the same way in PyTorch via device abstraction.

☁️

Free GPU options

Google Colab (T4 GPU, free tier), Kaggle Notebooks (P100, free), Paperspace (free T4). More than enough for learning all of Tier 3.

Pythondevice-agnostic PyTorch code

import torch
import torch.nn as nn

# ── Device setup — works on any machine ───────────────────────────────
if torch.cuda.is_available():
    device = torch.device("cuda")         # NVIDIA GPU
elif torch.backends.mps.is_available():
    device = torch.device("mps")          # Apple Silicon
else:
    device = torch.device("cpu")
print(f"Using device: {device}")

# ── Move model to device ───────────────────────────────────────────────
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 10)
).to(device)   # .to(device) moves ALL parameters

# ── Move data to device in the training loop ───────────────────────────
for X_batch, y_batch in train_loader:
    X_batch = X_batch.to(device)   # must match model device
    y_batch = y_batch.to(device)

    outputs = model(X_batch)       # runs on GPU
    # ... rest of training loop

# ── Checking GPU memory (useful for debugging OOM errors) ──────────────
if torch.cuda.is_available():
    print(torch.cuda.get_device_name(0))
    mem_alloc = torch.cuda.memory_allocated(0) / 1e9
    mem_total = torch.cuda.get_device_properties(0).total_memory / 1e9
    print(f"GPU memory: {mem_alloc:.2f} GB / {mem_total:.2f} GB")

# ── Saving and loading (always save to CPU-compatible format) ──────────
torch.save(model.state_dict(), "model.pt")

model2 = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
model2.load_state_dict(torch.load("model.pt", map_location=device))

⚠️

Out of Memory (OOM) errors The most common GPU error. Causes: batch size too large, model too large, not calling torch.no_grad() during eval (keeps computation graph in memory), or accidentally accumulating tensors in a list. Fix: reduce batch size first. Then check you're using with torch.no_grad() for inference. Use torch.cuda.empty_cache() to release cached memory.