Generative
Models
Networks that don't just classify — they create. From the simple idea of an autoencoder to the adversarial dynamics of GANs and the foundations of diffusion models that power modern image generation.
Autoencoders
An autoencoder learns to compress data into a lower-dimensional representation (encoding) and then reconstruct the original input from that representation (decoding). It's trained with a simple objective: minimize reconstruction error. No labels needed — it's a form of self-supervised learning.
x̂ = Decoder(z) ← reconstruct from the compressed representation
Loss = ||x − x̂||² ← reconstruction error (MSE, or BCE for images)
The bottleneck forces learning
If the latent dimension equals the input dimension, the network could trivially learn the identity function. The bottleneck (smaller latent dim) forces it to discover which features matter most for reconstruction — effectively learned, non-linear dimensionality reduction.
Use cases
Dimensionality reduction (nonlinear alternative to PCA), denoising (train on noisy input, clean target), anomaly detection (high reconstruction error = anomalous input), pretraining feature extractors.
Limitation: not generative
A vanilla autoencoder's latent space has no enforced structure. You can't sample a random z and decode it into something meaningful — the latent space has "holes" the decoder was never trained to handle. This motivates VAEs.
Denoising autoencoders
Add noise to the input, train the network to reconstruct the clean version. Forces the network to learn robust features rather than memorizing exact pixel values — a useful regularization technique.
import torch
import torch.nn as nn
class Autoencoder(nn.Module):
def __init__(self, input_dim=784, latent_dim=32):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256), nn.ReLU(),
nn.Linear(256, 64), nn.ReLU(),
nn.Linear(64, latent_dim) # bottleneck — no activation here
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 64), nn.ReLU(),
nn.Linear(64, 256), nn.ReLU(),
nn.Linear(256, input_dim),
nn.Sigmoid() # output in [0,1] for normalized pixel values
)
def forward(self, x):
z = self.encoder(x)
x_hat = self.decoder(z)
return x_hat, z
# Training loop sketch
model = Autoencoder(input_dim=784, latent_dim=32)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()
for x_batch in train_loader: # no labels needed!
x_flat = x_batch.view(x_batch.size(0), -1)
x_hat, z = model(x_flat)
loss = criterion(x_hat, x_flat) # reconstruct the input itself
optimizer.zero_grad(); loss.backward(); optimizer.step()Variational Autoencoders (VAEs)
VAEs (Kingma & Welling, 2013) fix the autoencoder's "holey latent space" problem by forcing the latent space to follow a known distribution (typically a standard Gaussian). This means you can sample any point from that distribution and decode it into a plausible output — VAEs are genuinely generative.
Sample: z = μ + σ ⊙ ε, where ε ~ N(0, I) ← "reparameterization trick"
Loss = Reconstruction_Loss + β · KL(N(μ,σ²) || N(0,I))
You can't backpropagate through a random sampling operation directly — sampling isn't differentiable. The trick: instead of sampling z directly from N(μ, σ²), sample noise ε from a fixed N(0,1) and compute z = μ + σ·ε. Now μ and σ are deterministic functions of the input (differentiable), and the randomness is isolated in ε (which doesn't need gradients).
import torch
import torch.nn as nn
import torch.nn.functional as F
class VAE(nn.Module):
def __init__(self, input_dim=784, latent_dim=20):
super().__init__()
self.fc1 = nn.Linear(input_dim, 256)
self.fc_mu = nn.Linear(256, latent_dim) # outputs mean
self.fc_logvar = nn.Linear(256, latent_dim) # outputs log-variance
self.fc2 = nn.Linear(latent_dim, 256)
self.fc3 = nn.Linear(256, input_dim)
def encode(self, x):
h = F.relu(self.fc1(x))
return self.fc_mu(h), self.fc_logvar(h)
def reparameterize(self, mu, logvar):
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std) # sample noise — no grad needed here
return mu + eps * std # reparameterization trick
def decode(self, z):
h = F.relu(self.fc2(z))
return torch.sigmoid(self.fc3(h))
def forward(self, x):
mu, logvar = self.encode(x)
z = self.reparameterize(mu, logvar)
return self.decode(z), mu, logvar
def vae_loss(x_hat, x, mu, logvar, beta=1.0):
recon_loss = F.binary_cross_entropy(x_hat, x, reduction='sum')
# KL divergence between N(mu, var) and N(0, 1) — closed form
kl_div = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return recon_loss + beta * kl_div
# Generate new samples after training
model = VAE()
model.eval()
with torch.no_grad():
z_random = torch.randn(16, 20) # sample from standard normal
generated = model.decode(z_random) # decode into new images!
print(generated.shape) # (16, 784) — 16 new generated samplesGANs — Generator/Discriminator Intuition
Generative Adversarial Networks (Goodfellow et al., 2014) take a radically different approach: two networks compete. A Generator creates fake samples trying to fool a Discriminator, which tries to distinguish real samples from fake ones. Through this adversarial competition, the generator learns to produce increasingly realistic outputs.
D(x) → probability that x is real (trained to maximize accuracy)
G(z) → generates fake sample from random noise z (trained to fool D)
- 1Train the Discriminator: Show it a batch of real images (label=1) and a batch of generator-produced fake images (label=0). Update D to better distinguish them.
- 2Train the Generator: Generate fake images from random noise, pass them through the (now frozen) discriminator, and update G to make D classify them as real.
- 3Alternate: Repeat steps 1 and 2. As D gets better at detecting fakes, G is forced to produce more realistic images to keep fooling it. Ideally they reach a Nash equilibrium where G produces samples indistinguishable from real data.
Training instability
GANs are notoriously difficult to train. If D becomes too strong too fast, G's gradients vanish (D always says "fake" with high confidence, no useful gradient). If G becomes too strong, D can't provide a useful learning signal. Balance is delicate.
Mode collapse
G discovers one or a few outputs that reliably fool D, and produces only those — losing diversity. E.g. a digit generator that only produces "1"s because that's easiest to fool D with. Active research area to prevent.
Sharp, realistic outputs
Unlike VAEs (which tend toward blurry reconstructions due to the pixel-wise reconstruction loss), GANs produce sharp, photorealistic images because the discriminator penalizes any perceptible blurriness or artifacts.
No explicit likelihood
Unlike VAEs, GANs don't optimize an explicit likelihood of the data. This makes evaluation harder — you can't compute a clean loss number to track "how good" the generator is. Metrics like FID (Fréchet Inception Distance) are used instead.
import torch
import torch.nn as nn
class Generator(nn.Module):
def __init__(self, noise_dim=100, img_dim=784):
super().__init__()
self.net = nn.Sequential(
nn.Linear(noise_dim, 256), nn.LeakyReLU(0.2),
nn.Linear(256, 512), nn.LeakyReLU(0.2),
nn.Linear(512, img_dim),
nn.Tanh() # output in [-1, 1]
)
def forward(self, z): return self.net(z)
class Discriminator(nn.Module):
def __init__(self, img_dim=784):
super().__init__()
self.net = nn.Sequential(
nn.Linear(img_dim, 512), nn.LeakyReLU(0.2),
nn.Linear(512, 256), nn.LeakyReLU(0.2),
nn.Linear(256, 1),
nn.Sigmoid() # probability of being real
)
def forward(self, x): return self.net(x)
G, D = Generator(), Discriminator()
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
criterion = nn.BCELoss()
for real_imgs in train_loader:
batch_size = real_imgs.size(0)
real_labels = torch.ones(batch_size, 1)
fake_labels = torch.zeros(batch_size, 1)
# ── Train Discriminator ──────────────────────────────────────────
opt_D.zero_grad()
real_loss = criterion(D(real_imgs), real_labels)
z = torch.randn(batch_size, 100)
fake_imgs = G(z)
fake_loss = criterion(D(fake_imgs.detach()), fake_labels) # detach: don't update G here
d_loss = real_loss + fake_loss
d_loss.backward(); opt_D.step()
# ── Train Generator ──────────────────────────────────────────────
opt_G.zero_grad()
g_loss = criterion(D(fake_imgs), real_labels) # wants D to think these are real
g_loss.backward(); opt_G.step().detach() on the generator's output. Otherwise, the discriminator's backward pass would also compute gradients through the generator, polluting its training. The generator should only receive gradients during its own training step.Diffusion Models (Conceptual Introduction)
Diffusion models are the architecture behind DALL-E 2/3, Stable Diffusion, and Midjourney. They take a fundamentally different approach from GANs and VAEs: instead of generating an image in one shot, they gradually denoise random noise into a coherent image over many steps. Full technical depth is covered in Tier 4; this is the conceptual foundation.
- 1Forward process (fixed, not learned): Take a real image and gradually add Gaussian noise over T steps (e.g. T=1000) until it becomes pure noise. This process is mathematically defined, not trained.
- 2Reverse process (learned): Train a neural network to predict and remove the noise added at each step — essentially learning to denoise. Given a noisy image at step t, predict what noise was added to get there.
- 3Generation: Start from pure random noise. Apply the learned denoising network repeatedly, step by step, gradually transforming noise into a coherent image. Each step removes a small amount of predicted noise.
Training objective: minimize ||ε − ε_θ(xₜ, t)||²
(predict the noise that was added, given the noisy image and timestep)
Slower generation than GANs
GANs generate in one forward pass. Diffusion models require many denoising steps (originally 1000, now optimized to 20-50 with techniques like DDIM). This makes diffusion slower but more stable to train and higher quality.
Training stability
Unlike GANs' adversarial dynamics (which can collapse or oscillate), diffusion training is a straightforward regression problem — predict the noise. Much more stable and reproducible to train.
Diversity
Diffusion models don't suffer from mode collapse the way GANs do — they tend to capture the full diversity of the training distribution, which is part of why they produce such varied, high-quality outputs.
Conditioning (text-to-image)
Modern diffusion models (Stable Diffusion, DALL-E) condition the denoising process on text embeddings (from a model like CLIP), guiding generation toward images matching a text prompt — this is how "a cat astronaut" becomes a coherent image.