Reinforcement
Learning Basics

Reinforcement learning (RL) is a fundamentally different paradigm from everything in this tier so far. Supervised learning trains on (input, correct-output) pairs. RL trains an agent that takes actions in an environment, receives rewards, and learns a policy — a strategy for choosing actions — that maximizes cumulative reward over time, without ever being told the "correct" action directly.

A core tension in RL: should the agent exploit what it already knows works well, or explore to discover potentially better strategies? Always exploiting risks getting stuck in a mediocre local strategy. Always exploring wastes opportunities to use known-good actions. Balancing these is a central RL design challenge, often handled with strategies like ε-greedy (mostly exploit, occasionally explore randomly) or more sophisticated approaches.

The Markov Decision Process (MDP) is the formal mathematical framework underlying almost all of RL. It defines precisely what "states," "actions," "rewards," and "transitions" mean, and crucially makes the Markov assumption: the future depends only on the current state, not on the history of how you got there.

The agent's goal is to maximize the return — the discounted sum of all future rewards. The discount factor γ controls how much to value future rewards versus immediate ones.

Q-learning is a classic, foundational RL algorithm that learns the optimal action-value function Q*(s,a) directly from experience, without needing to know the environment's transition probabilities in advance (model-free learning). It uses the Bellman equation as an iterative update rule.

• 1Initialize a Q-table with one entry per (state, action) pair, typically zeros.
• 2Observe the current state s.
• 3Choose an action using ε-greedy: with probability ε pick a random action (explore), otherwise pick argmax_a Q(s,a) (exploit).
• 4Execute the action, observe reward R and new state s'.
• 5Update Q(s,a) using the Bellman update equation above.
• 6Repeat until convergence — Q-values stabilize toward the true optimal values.

import numpy as np

# Simplified grid world: 4 states, 2 actions (left/right)
n_states, n_actions = 4, 2
Q = np.zeros((n_states, n_actions))

# Simulated environment transition (toy example)
def step(state, action):
    # action 0 = left, action 1 = right
    next_state = max(0, state - 1) if action == 0 else min(n_states-1, state + 1)
    reward = 1.0 if next_state == n_states - 1 else 0.0   # goal at last state
    done = (next_state == n_states - 1)
    return next_state, reward, done

# ── Q-learning hyperparameters ───────────────────────────────────────
alpha, gamma, epsilon = 0.1, 0.95, 1.0
epsilon_decay = 0.995

for episode in range(1000):
    state = 0
    done = False
    while not done:
        # ε-greedy action selection
        if np.random.rand() < epsilon:
            action = np.random.randint(n_actions)   # explore
        else:
            action = np.argmax(Q[state])              # exploit

        next_state, reward, done = step(state, action)

        # Bellman update — the core of Q-learning
        best_next_q = np.max(Q[next_state])
        td_target = reward + gamma * best_next_q
        td_error  = td_target - Q[state, action]
        Q[state, action] += alpha * td_error

        state = next_state

    epsilon = max(0.01, epsilon * epsilon_decay)   # decay exploration over time

print("Learned Q-table:")
print(Q.round(3))
# Optimal policy: argmax over each row → should favor "right" (action 1)

Q-learning learns values and derives a policy indirectly (pick the action with the highest Q-value). Policy gradient methods take a more direct approach: parameterize the policy itself as a neural network, and directly optimize its parameters to maximize expected reward using gradient ascent.

Reinforcement Learning from Human Feedback (RLHF) is the technique that turns a raw pretrained language model (good at predicting next tokens) into an assistant that follows instructions, avoids harmful outputs, and feels genuinely helpful. It's RL applied to a domain very different from games or robotics: the "environment" is a conversation, and the "reward" comes from human preferences.

• 1Supervised Fine-Tuning (SFT): Start with a pretrained language model. Fine-tune it on a dataset of high-quality human-written demonstrations (prompt → ideal response). This gives the model a baseline sense of "what a good response looks like."
• 2Reward Model training: Collect human preference data — pairs of model responses where a human indicates which is better. Train a separate neural network (the reward model) to predict this human preference score for any given response.
• 3RL fine-tuning (typically PPO): Use the reward model as the reward signal in an RL loop. The language model (now the "policy") generates responses, the reward model scores them, and PPO updates the language model's weights to generate higher-scoring responses — while a KL-divergence penalty keeps it from drifting too far from the SFT model (preventing reward hacking and gibberish).