Neural Networks
From Scratch
Before you use a framework, you build it yourself. This module constructs neural networks from first principles — from the single perceptron to a multi-layer net in raw NumPy — so the abstractions you reach for later are never magic.
The Perceptron — The Original Neural Network
Every neural network you'll ever build — GPT, ResNet, DALL-E — is a descendant of a remarkably simple idea proposed by Frank Rosenblatt in 1957: the perceptron. It's worth understanding it completely, not as a historical footnote, but as the conceptual atom that everything else compounds from.
A perceptron takes a vector of inputs, multiplies each by a learned weight, sums everything up, adds a bias, and passes the result through a threshold function. That's the complete description.
= f( w · x + b )
The weights w are what the network "learns." The bias b shifts the decision boundary. The dot product w · x is a weighted vote from every input feature. The step function makes a hard binary decision.
Inputs (x)
Feature values from your data — pixel intensities, sensor readings, encoded categories. Each input connects to the perceptron with its own wire.
Weights (w)
The strength and sign of each connection. A large positive weight means "this input strongly votes yes." A negative weight votes no.
Bias (b)
A trainable offset that shifts where the decision boundary sits, independent of the inputs. Without it, the boundary is forced through the origin.
Activation (f)
The function applied to the weighted sum. In the original perceptron, a step function. In modern nets, something differentiable (ReLU, sigmoid — next lesson).
The perceptron doesn't just classify — it learns. The update rule is simple: if the perceptron gets an example wrong, nudge the weights in the direction that would have produced the right answer.
If the prediction is correct (y = ŷ), the update is zero — nothing changes. If wrong, every weight shifts by an amount proportional to the corresponding input. This is the seed of backpropagation, which you'll see in Module 2.
import numpy as np
class Perceptron:
def __init__(self, n_features, lr=0.01):
# Small random weights, one per feature plus a bias
self.weights = np.random.randn(n_features) * 0.01
self.bias = 0.0
self.lr = lr
def predict(self, X):
# Weighted sum + bias, then step function
linear = np.dot(X, self.weights) + self.bias
return (1 if linear >= 0 else 0)
def train(self, X, y, epochs=100):
for _ in range(epochs):
for xi, yi in zip(X, y):
pred = self.predict(xi)
error = yi - pred # 0 if correct, ±1 if wrong
# Update: push weights toward the right answer
self.weights += self.lr * error * xi
self.bias += self.lr * error
# Simple AND gate — the classic perceptron demo
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0, 0, 0, 1]) # AND: only 1 when both inputs are 1
p = Perceptron(n_features=2)
p.train(X, y, epochs=200)
for xi in X:
print(f"{xi} → {p.predict(xi)}")
# [0 0] → 0 [0 1] → 0 [1 0] → 0 [1 1] → 1 ✓A single perceptron draws one decision boundary. Stack several perceptrons in parallel (a layer), and each one draws its own boundary. Stack layers, and the network can learn arbitrarily complex boundaries — it approximates any continuous function given enough neurons. This is the Universal Approximation Theorem, and it's why deep networks work.
Activation Functions: Sigmoid, Tanh, ReLU, and Why ReLU Mostly Won
The activation function is what makes a neural network non-linear — and non-linearity is what makes it capable of learning anything beyond a linear relationship. Without activations, stacking layers achieves nothing: a linear function composed with a linear function is still a linear function.
| Function | Formula | Range | Gradient | Use Today |
|---|---|---|---|---|
| Sigmoid σ(x) | 1 / (1 + e⁻ˣ) |
(0, 1) | σ(x)(1 - σ(x)) — small near extremes | Output layer only |
| Tanh | (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ) |
(-1, 1) | 1 - tanh²(x) — same vanishing problem | RNNs, some gates |
| ReLU | max(0, x) |
[0, ∞) | 1 if x>0, else 0 — sparse and fast | Default for hidden layers |
Sigmoid: historically first
Maps any input to (0,1), great for probability outputs. Fatal flaw: at large positive or negative inputs, the gradient is nearly zero — gradients "vanish" as they flow backward through layers. Deep networks with sigmoid can't train.
Tanh: sigmoid's improvement
Zero-centered (outputs range -1 to 1), which helps gradient flow compared to sigmoid. Still suffers from saturation at extremes. Preferred over sigmoid for hidden layers when it was all we had, still used in LSTMs.
ReLU: the modern default
Embarrassingly simple: zero below zero, identity above. The gradient is always either 0 or 1 — no saturation for positive inputs. Training deep networks became practical once ReLU replaced sigmoid.
ReLU variants
Leaky ReLU (small slope for x<0 to fix "dead neurons"), GELU (used in transformers — smooth approximation of ReLU), Swish (GELU alternative). ReLU is still the default unless you have a specific reason.
Three concrete reasons ReLU enabled the deep learning revolution:
- 1No vanishing gradient for active neurons. When x > 0, the gradient is exactly 1. It doesn't shrink as it flows through layers. Sigmoid's gradient maxes out at 0.25 — compound that across 20 layers and you get ~10⁻⁶. No signal reaches early layers.
- 2Sparse activation. On average, ~50% of ReLU neurons output zero. This sparsity means networks are computationally efficient and representations are compact — different inputs activate different subsets of the network.
- 3Cheap to compute. max(0, x) is one comparison instruction. No exponentials, no divisions. When you're running billions of these operations, that matters enormously.
import numpy as np
# Sigmoid and its gradient
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def sigmoid_grad(x):
s = sigmoid(x)
return s * (1 - s) # Max 0.25 at x=0
# Tanh and its gradient
def tanh(x):
return np.tanh(x)
def tanh_grad(x):
return 1 - np.tanh(x)**2 # Max 1.0 at x=0
# ReLU and its gradient
def relu(x):
return np.maximum(0, x)
def relu_grad(x):
return (x > 0).astype(float) # 1 where active, 0 where not
# Leaky ReLU — fixes "dying ReLU" by allowing small gradient at x<0
def leaky_relu(x, alpha=0.01):
return np.where(x > 0, x, alpha * x)
# Quick comparison at x = [-2, -1, 0, 1, 2]
x = np.array([-2., -1., 0., 1., 2.])
print("ReLU: ", relu(x)) # [0. 0. 0. 1. 2.]
print("Sigmoid:", sigmoid(x).round(2)) # [0.12 0.27 0.5 0.73 0.88]
print("Tanh: ", np.tanh(x).round(2)) # [-0.96 -0.76 0. 0.76 0.96]Forward Propagation, By Hand, On Paper First
Before writing code, do it on paper. Forward propagation is a mechanical process: transform inputs through layer after layer until you get an output. Running it by hand once — with real numbers — means you'll never be confused about what "a layer" actually does.
model(x) runs forward propagation invisibly. If your output is wrong, you need to be able to trace the computation to find the bug. People who skipped the manual step can't do this effectively.We have a network with 2 inputs, 3 hidden neurons, and 1 output. Let's trace a single forward pass with specific numbers.
Layer 1 weights W¹ = [[0.1, 0.4], [0.3, -0.2], [-0.1, 0.5]]
Layer 1 biases b¹ = [0, 0, 0]
Layer 2 weights W² = [[0.6, -0.3, 0.8]]
Layer 2 bias b² = [0]
- 1Linear combination, Layer 1: z¹ = W¹x + b¹
z¹₁ = 0.1(0.5) + 0.4(0.8) = 0.05 + 0.32 = 0.37
z¹₂ = 0.3(0.5) + (-0.2)(0.8) = 0.15 − 0.16 = −0.01
z¹₃ = −0.1(0.5) + 0.5(0.8) = −0.05 + 0.40 = 0.35 - 2Apply ReLU, Layer 1: a¹ = ReLU(z¹)
a¹ = ReLU([0.37, −0.01, 0.35]) = [0.37, 0.00, 0.35]
The second neuron is "off" — its weighted sum was negative. - 3Linear combination, Layer 2: z² = W²a¹ + b²
z² = 0.6(0.37) + (−0.3)(0.00) + 0.8(0.35) = 0.222 + 0 + 0.280 = 0.502 - 4Apply sigmoid, output layer: ŷ = σ(0.502) = 1/(1+e⁻⁰·⁵⁰²) ≈ 0.623
Interpretation: the network predicts a 62.3% probability of class 1.
import numpy as np
# Same values as our worked example
x = np.array([0.5, 0.8])
W1 = np.array([[0.1, 0.4],
[0.3, -0.2],
[-0.1, 0.5]])
b1 = np.zeros(3)
W2 = np.array([[0.6, -0.3, 0.8]])
b2 = np.zeros(1)
# Layer 1
z1 = W1 @ x + b1 # Matrix multiply + bias
a1 = np.maximum(0, z1) # ReLU
print("z1:", z1.round(4)) # [ 0.37 -0.01 0.35]
print("a1:", a1.round(4)) # [ 0.37 0. 0.35]
# Layer 2
z2 = W2 @ a1 + b2
print("z2:", z2.round(4)) # [0.502]
# Output with sigmoid
y_hat = 1 / (1 + np.exp(-z2))
print("ŷ: ", y_hat.round(4)) # [0.623] ← matches hand calc ✓Pre-activation (z)
The raw weighted sum before the activation function. Also called the "logit." The activation function transforms z into the neuron's output.
Activation (a)
The output of a neuron after applying the activation function. a = f(z). The activations from one layer become the inputs to the next.
Weight matrix (W)
A matrix of shape (output_neurons × input_neurons). The matrix multiply W@x computes all neurons in a layer simultaneously — this is why vectorization matters.
Layer index (ℓ)
Superscripts denote layers: W^[1] is layer 1's weights, a^[2] is layer 2's activations. The input is a^[0] = x. The output is a^[L] = ŷ.
Building a Tiny Neural Net in Raw NumPy
This is the lesson that matters most in this module, even though it's the last. You're going to build a working neural network — training included — using only NumPy. No PyTorch, no Keras, no scikit-learn. Just math and arrays.
When you do this, the things that feel like magic in frameworks become engineering decisions you understand completely: how weights are initialized, how gradients flow, how the training loop works. You'll also make mistakes — divide by the wrong dimension, forget the chain rule on a particular layer — and that confusion is the most valuable debugging training you can get before the abstractions hide it.
import numpy as np
# ─── Data ───────────────────────────────────────────────────────────────
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float) # (4, 2)
y = np.array([[0],[1],[1],[0]], dtype=float) # (4, 1)
# ─── Architecture ───────────────────────────────────────────────────────
np.random.seed(42)
n_input, n_hidden, n_output = 2, 4, 1
# Xavier / Glorot init: keeps variance stable through layers
W1 = np.random.randn(n_input, n_hidden) * np.sqrt(2 / n_input)
b1 = np.zeros((1, n_hidden))
W2 = np.random.randn(n_hidden, n_output) * np.sqrt(2 / n_hidden)
b2 = np.zeros((1, n_output))
lr = 0.1
# ─── Training loop ──────────────────────────────────────────────────────
for epoch in range(10000):
# ── Forward pass ──────────────────────────────────────────────────
z1 = X @ W1 + b1 # (4, 4) linear combination
a1 = np.maximum(0, z1) # (4, 4) ReLU activation
z2 = a1 @ W2 + b2 # (4, 1) output layer linear
a2 = 1 / (1 + np.exp(-z2)) # (4, 1) sigmoid → prediction
# ── Loss: binary cross-entropy ─────────────────────────────────────
eps = 1e-8 # prevent log(0)
loss = -np.mean(y * np.log(a2 + eps) + (1-y) * np.log(1-a2 + eps))
# ── Backward pass (backpropagation) ────────────────────────────────
# Output layer gradients
dL_da2 = -(y/(a2+eps) - (1-y)/(1-a2+eps)) # dLoss/da2
da2_dz2 = a2 * (1 - a2) # sigmoid derivative
dz2 = dL_da2 * da2_dz2 # (4, 1)
dW2 = a1.T @ dz2 / len(X) # (4, 1)
db2 = np.mean(dz2, axis=0) # (1, 1)
# Hidden layer gradients — chain rule through W2 and ReLU
da1 = dz2 @ W2.T # (4, 4) gradient flows back through W2
dz1 = da1 * (z1 > 0) # (4, 4) ReLU gradient (0 where inactive)
dW1 = X.T @ dz1 / len(X) # (2, 4)
db1 = np.mean(dz1, axis=0) # (1, 4)
# ── Gradient descent update ────────────────────────────────────────
W2 -= lr * dW2; b2 -= lr * db2
W1 -= lr * dW1; b1 -= lr * db1
if epoch % 2000 == 0:
print(f"Epoch {epoch:5d} loss: {loss:.4f}")
# ─── Verify ─────────────────────────────────────────────────────────────
preds = (1 / (1 + np.exp(-(a1 @ W2 + b2)))) > 0.5
print("\nXOR predictions:")
for xi, yi, pi in zip(X, y, preds):
print(f" {xi} → true:{int(yi[0])} pred:{int(pi[0])}")
# All four XOR cases should now be correct — what the perceptron couldn't dodz2 = dL_da2 * da2_dz2 is the chain rule: derivative of loss w.r.t. z2 = (dL/da2) × (da2/dz2).Xavier initialization
Weights initialized with variance 2/n_inputs keeps signal magnitude stable through layers. Bad initialization → vanishing or exploding gradients before training even starts.
Binary cross-entropy loss
The right loss for binary classification. It penalizes confident wrong answers heavily and doesn't saturate the way MSE does with sigmoid outputs.
Vectorized operations
All four training examples computed simultaneously with matrix operations — no Python for-loop over examples. This is why GPUs are fast: they run thousands of these in parallel.
Gradient descent update
Weights shift opposite the gradient direction. The learning rate lr controls the step size. This is the same algorithm that trains GPT, just at 10⁹× the scale.