Tier 3 · Deep Learning & Neural Networks

Recurrent Networks
& Sequence Models

Language, audio, time series — ordered data where what came before shapes what comes next. RNNs, LSTMs, and GRUs process sequences by threading a hidden state through time. You'll understand their mechanics, their failure modes, and build a text generator from scratch.

📚 5 Lessons 📝 NLP Focus ✍️ Text Gen Project ⏱ ~4 hours

Lesson 6.1

Why Sequences Need a Different Architecture Than Images

Images have fixed, known structure: width × height × channels. Every image in a dataset can be the same size. Every pixel position has a consistent meaning relative to its neighbors. MLPs and CNNs exploit this rigidity.

Sequences break all of these assumptions. A sentence can be 3 words or 300 words. The meaning of a word depends on all the words before it. The word "bank" means something completely different in "river bank" vs "savings bank." These two sentences have opposite meanings: "The food was good, not bad at all" and "The food was bad, not good at all." Processing them correctly requires understanding order, context, and variable length simultaneously.

📏

Variable length

Sequences differ in length. A fixed-size MLP can't handle this. The architecture must process arbitrary-length inputs without changing its parameter count — which is exactly what RNNs do.

⏱

Temporal dependencies

Earlier elements influence the meaning of later elements. To predict the next word in "The cat sat on the ___", you need to remember "cat sat on the", not just "the". Memory across time is essential.

🔄

Weight sharing across time

Just as CNNs share weights across spatial positions, RNNs share weights across time steps. The same transformation is applied at each time step — the network learns one function to apply repeatedly.

📤

Output flexibility

Sequences enable many-to-many (translation), many-to-one (sentiment), one-to-many (image captioning), and one-to-one mappings — more output structures than images typically need.

🔍

The three main architectures for sequences RNNs (vanilla, now mostly replaced) → LSTMs/GRUs (gated, handle longer dependencies, still used) → Transformers (attention-based, parallel, currently dominant for NLP). This module covers the first two in depth; Module 7–8 cover transformers.

Lesson 6.2

RNNs and the Vanishing Gradient Problem in Sequences

A vanilla RNN processes sequences by maintaining a hidden state — a vector that summarizes everything the network has seen so far. At each time step, it takes the current input and the previous hidden state, runs them through a linear transformation and activation, and produces a new hidden state.

hₜ = tanh(Wₕ \cdot hₜ₋₁ + Wₓ \cdot xₜ + b) ŷₜ = Wᵧ \cdot hₜ + bᵧ Same Wₕ, Wₓ at every time step — weight sharing across time hₜ is the hidden state at step t. It's both the output and the memory.

The vanishing gradient problem — again, but worse

You saw vanishing gradients in Module 2 for depth. In RNNs the problem is worse: backpropagation through time (BPTT) unrolls the recurrence, computing gradients through every time step. For a sequence of length T, the gradient at step 1 involves multiplying the same weight matrix T times:

\partialL/\partialh₁ = \partialL/\partialhₜ \cdot \prodᵢ₌₂ᵀ (\partialhᵢ/\partialhᵢ₋₁) Each factor involves Wₕ \cdot diag(tanh'(...)) If ||Wₕ|| < 1: gradients vanish exponentially with T If ||Wₕ|| > 1: gradients explode exponentially with T For T=100 (a short sentence), this is multiplying 100 matrices together

💀

Practical consequence

Vanilla RNNs can only learn short-range dependencies — typically 5–10 time steps. For longer contexts, the gradient signal from early steps is too small to update the weights meaningfully. The network forgets.

✂️

Truncated BPTT

Practical fix: don't backpropagate all the way through the full sequence. Backpropagate only through the last K steps (e.g. K=20). Cheaper computation, but the network still can't learn very long-range dependencies.

📎

Gradient clipping

Prevents exploding gradients. Standard for all RNN training. Clip gradient norm to a maximum value (e.g. 5.0) before the parameter update. Doesn't fix vanishing — just prevents explosion.

🏗️

The real fix: gating

The fundamental solution is LSTMs and GRUs, which use learned gates to control what information flows through time — effectively creating direct paths for gradients to flow over long sequences.

Pythonvanilla RNN in PyTorch

import torch
import torch.nn as nn

# nn.RNN: input_size=features per step, hidden_size=state dim
rnn = nn.RNN(input_size=10, hidden_size=64, num_layers=2,
              batch_first=True,   # input shape: (batch, seq_len, features)
              dropout=0.2)         # dropout between layers

# Input: batch=8, sequence length=20, 10 features per step
x = torch.randn(8, 20, 10)
h0 = torch.zeros(2, 8, 64)   # 2 layers, batch=8, hidden=64

output, hn = rnn(x, h0)
print("Output shape:", output.shape)  # (8, 20, 64) — hidden state at every step
print("Final hidden:", hn.shape)      # (2, 8, 64)  — last hidden state, both layers

# For sequence classification: use only the last time step's output
last_output = output[:, -1, :]     # (8, 64) — final state summarizes the sequence
classifier = nn.Linear(64, 2)      # e.g. sentiment: positive/negative
logits = classifier(last_output)   # (8, 2)

Lesson 6.3

LSTMs and GRUs

The Long Short-Term Memory (LSTM, Hochreiter & Schmidhuber 1997) introduced a second state vector — the cell state — that carries information across long sequences with minimal transformation. Learned gates control what information is written, read, and erased from the cell state, allowing gradients to flow through time without vanishing.

LSTM: four gates, two states

fₜ = σ(Wf \cdot [hₜ₋₁, xₜ] + bf) \leftarrow forget gate: what to erase from cell iₜ = σ(Wi \cdot [hₜ₋₁, xₜ] + bi) \leftarrow input gate: what new info to write c̃ₜ = tanh(Wc \cdot [hₜ₋₁, xₜ] + bc) \leftarrow candidate cell values cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ \leftarrow update cell state oₜ = σ(Wo \cdot [hₜ₋₁, xₜ] + bo) \leftarrow output gate: what to reveal hₜ = oₜ ⊙ tanh(cₜ) \leftarrow hidden state ⊙ = element-wise multiply. σ = sigmoid (outputs 0-1, acts as a soft gate)

The key insight is the cell state update: cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ. This is an additive update — gradients flow back through addition (gradient of addition is 1), not multiplication. The forget gate learns when to truly erase information; until then, the cell state is approximately preserved across many steps.

GRU: simplified LSTM

The Gated Recurrent Unit (GRU, Cho et al. 2014) merges the cell state and hidden state into one, and simplifies to two gates. Fewer parameters, similar performance to LSTM on most tasks, faster to train.

zₜ = σ(Wz \cdot [hₜ₋₁, xₜ]) \leftarrow update gate (combines forget + input) rₜ = σ(Wr \cdot [hₜ₋₁, xₜ]) \leftarrow reset gate (controls past influence) h̃ₜ = tanh(W \cdot [rₜ⊙hₜ₋₁, xₜ]) \leftarrow candidate hidden state hₜ = (1-zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ \leftarrow interpolate old and new GRU has ~25% fewer parameters than LSTM of the same hidden size

Model	States	Gates	Parameters	Use when
Vanilla RNN	1 (h)	0	Fewest	Avoid — vanishing gradient
LSTM	2 (h, c)	3	4× RNN	Long sequences, complex tasks
GRU	1 (h)	2	3× RNN	Default gated RNN choice

PythonLSTM and GRU in PyTorch — API comparison

import torch
import torch.nn as nn

x = torch.randn(8, 50, 32)   # batch=8, seq_len=50, features=32

# LSTM: returns (output, (h_n, c_n)) — hidden AND cell state
lstm = nn.LSTM(input_size=32, hidden_size=128, num_layers=2,
               batch_first=True, dropout=0.2, bidirectional=False)
out_lstm, (hn, cn) = lstm(x)
print("LSTM output:", out_lstm.shape)   # (8, 50, 128)
print("LSTM h_n:   ", hn.shape)         # (2, 8, 128)  ← 2 layers
print("LSTM c_n:   ", cn.shape)         # (2, 8, 128)  ← cell state

# GRU: returns (output, h_n) — only one state
gru = nn.GRU(input_size=32, hidden_size=128, num_layers=2,
              batch_first=True, dropout=0.2)
out_gru, hn_gru = gru(x)
print("GRU output: ", out_gru.shape)    # (8, 50, 128)
print("GRU h_n:    ", hn_gru.shape)     # (2, 8, 128)

# Bidirectional LSTM: processes sequence in both directions
# Doubles output features: hidden_size * 2
bilstm = nn.LSTM(32, 64, batch_first=True, bidirectional=True)
out_bi, _ = bilstm(x)
print("BiLSTM:     ", out_bi.shape)     # (8, 50, 128)  ← 64*2=128

✅

When to use bidirectional A bidirectional RNN/LSTM reads the sequence both forward and backward, giving each position context from both directions. Use for tasks where the full sequence is available at inference time (text classification, NER, machine translation encoder). Don't use for causal/autoregressive tasks (language modeling, real-time speech) where future tokens aren't available.

Lesson 6.4

Sequence-to-Sequence Models

Many real problems map one sequence to another of different length: machine translation (English → French), summarization (article → summary), speech recognition (audio → text). The seq2seq framework — encoder-decoder — handles this elegantly and established many patterns that transformers later adopted.

Encoder-Decoder architecture

1Encoder: An RNN (usually LSTM) reads the input sequence and compresses it into a fixed-size context vector — the final hidden state. This vector is the encoder's "summary" of the entire input.
2Context vector: The encoder's final hidden state. It's passed to the decoder as the initial hidden state, seeding the decoder with knowledge of the input sequence.
3Decoder: Another RNN that generates the output sequence one token at a time. At each step it receives the previously generated token and the current hidden state, and predicts the next token.
4Teacher forcing: During training, feed the correct previous token rather than the predicted one. This stabilizes training — errors don't compound. During inference, the predicted token is fed in (which can cause train/test mismatch).

⚠️

The bottleneck problem The entire input sequence — regardless of length — must be compressed into a single fixed-size vector. For long inputs this is lossy. A 500-word article compressed into a 256-dimensional vector loses information. This is the limitation that motivated attention mechanisms — instead of compressing everything into one vector, attend to different parts of the input at each decoder step.

Pythonseq2seq encoder-decoder in PyTorch

import torch
import torch.nn as nn

class Encoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, n_layers, dropout):
        super().__init__()
        self.embed  = nn.Embedding(vocab_size, embed_dim)
        self.rnn    = nn.LSTM(embed_dim, hidden_dim, n_layers,
                               batch_first=True, dropout=dropout)
        self.dropout = nn.Dropout(dropout)

    def forward(self, src):
        embedded = self.dropout(self.embed(src))
        _, (hidden, cell) = self.rnn(embedded)
        return hidden, cell   # the context: summarizes the source

class Decoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, n_layers, dropout):
        super().__init__()
        self.embed    = nn.Embedding(vocab_size, embed_dim)
        self.rnn      = nn.LSTM(embed_dim, hidden_dim, n_layers,
                                 batch_first=True, dropout=dropout)
        self.fc_out   = nn.Linear(hidden_dim, vocab_size)
        self.dropout  = nn.Dropout(dropout)

    def forward(self, token, hidden, cell):
        token = token.unsqueeze(1)            # (batch,) → (batch, 1)
        embedded = self.dropout(self.embed(token))  # (batch, 1, embed_dim)
        output, (hidden, cell) = self.rnn(embedded, (hidden, cell))
        prediction = self.fc_out(output.squeeze(1))  # (batch, vocab_size)
        return prediction, hidden, cell

class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder, teacher_force_ratio=0.5):
        super().__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.tfr     = teacher_force_ratio

    def forward(self, src, trg):
        batch_size, trg_len = trg.shape
        vocab_size = self.decoder.fc_out.out_features
        outputs = torch.zeros(batch_size, trg_len, vocab_size).to(src.device)

        hidden, cell = self.encoder.forward(src)
        token = trg[:, 0]   # start token

        for t in range(1, trg_len):
            out, hidden, cell = self.decoder.forward(token, hidden, cell)
            outputs[:, t] = out
            teacher_force = torch.rand(1) < self.tfr
            token = trg[:, t] if teacher_force else out.argmax(1)

        return outputs

Lesson 6.5

Project: Text Generation with an RNN

The capstone for this module is a character-level text generator: train an LSTM on a text corpus, then sample from it to produce new text in the style of the training data. This is the classic RNN project, used in Karpathy's famous "The Unreasonable Effectiveness of RNNs." Simple setup, but it demonstrates everything: tokenization, sequence batching, teacher forcing, and temperature sampling.

🎯

Learning objectives You'll implement character tokenization, create overlapping sequence batches, train an LSTM language model, and write a temperature-controlled sampling function. Feed it Shakespeare, Python code, or any text and watch it learn to imitate the style.

Pythoncharacter-level text generator — complete implementation

import torch
import torch.nn as nn
import numpy as np

# ── 1. Prepare data ───────────────────────────────────────────────────
text = open('input.txt').read()   # any text file
chars = sorted(set(text))
char2idx = {c: i for i, c in enumerate(chars)}
idx2char = {i: c for c, i in char2idx.items()}
vocab_size = len(chars)

data = torch.tensor([char2idx[c] for c in text], dtype=torch.long)
print(f"Vocab: {vocab_size} chars, Data: {len(data):,} tokens")

# ── 2. Batch creation ─────────────────────────────────────────────────
def get_batch(data, seq_len=100, batch_size=32):
    starts = torch.randint(len(data) - seq_len - 1, (batch_size,))
    X = torch.stack([data[s:s+seq_len]   for s in starts])
    Y = torch.stack([data[s+1:s+seq_len+1] for s in starts])
    return X, Y   # target is input shifted by 1: predict next char

# ── 3. Model ─────────────────────────────────────────────────────────
class CharLM(nn.Module):
    def __init__(self, vocab_size, embed_dim=64, hidden_dim=256, n_layers=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm  = nn.LSTM(embed_dim, hidden_dim, n_layers,
                               batch_first=True, dropout=0.3)
        self.head  = nn.Linear(hidden_dim, vocab_size)

    def forward(self, x, hidden=None):
        emb = self.embed(x)
        out, hidden = self.lstm(emb, hidden)
        logits = self.head(out)
        return logits, hidden

# ── 4. Training ───────────────────────────────────────────────────────
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = CharLM(vocab_size).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
loss_fn = nn.CrossEntropyLoss()

for step in range(5000):
    X, Y = get_batch(data)
    X, Y = X.to(device), Y.to(device)
    logits, _ = model(X)
    loss = loss_fn(logits.reshape(-1, vocab_size), Y.reshape(-1))
    opt.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # clip!
    opt.step()
    if step % 500 == 0: print(f"step {step}: loss {loss.item():.3f}")

# ── 5. Temperature sampling ────────────────────────────────────────────
def generate(model, seed_char, length=500, temperature=0.8):
    model.eval()
    token = torch.tensor([[char2idx[seed_char]]]).to(device)
    hidden = None
    result = [seed_char]
    with torch.no_grad():
        for _ in range(length):
            logits, hidden = model(token, hidden)
            logits = logits[0, 0] / temperature   # scale by temperature
            probs = torch.nn.functional.softmax(logits, dim=-1)
            next_token = torch.multinomial(probs, 1)   # sample, not argmax
            result.append(idx2char[next_token.item()])
            token = next_token.unsqueeze(0)
    return ''.join(result)

print(generate(model, seed_char='T', temperature=0.8))

✅

Temperature controls creativity Temperature < 1.0: sharper distribution → more predictable, repetitive text. Temperature = 1.0: sample from the raw model distribution. Temperature > 1.0: flatter distribution → more diverse, sometimes incoherent text. Try 0.5, 0.8, 1.0, 1.2 and compare the outputs. Low temp writes like a student memorizing; high temp writes like a drunk poet.