RL Fundamentals

Before diving into deep neural networks and RLHF, you need to understand the mathematical foundation of reinforcement learning: Markov Decision Processes, value functions, and the Bellman equations. These concepts underpin every RL algorithm from Q-learning to PPO.

Markov Decision Processes (MDPs)

Every RL problem is formally described as an MDP, defined by the tuple (S, A, P, R, γ):

The Markov property: the future depends only on the current state, not the history. P(sₜ₊₁ | sₜ, aₜ) = P(sₜ₊₁ | s₀, a₀, ..., sₜ, aₜ). This assumption makes the math tractable.

Value Functions

Value functions estimate how good it is to be in a state (or take an action in a state) when following a particular policy π:

The Bellman Equations

The Bellman equations express value functions recursively — the value of a state equals immediate reward plus discounted value of next states. This recursive structure is the foundation of virtually every RL algorithm.

The optimal policy can be derived from V*: at each state, pick the action that maximises expected return. The challenge: we rarely know P(s'|s,a) in real problems — we must estimate V* from experience.

Temporal Difference Learning

TD learning estimates value functions by bootstrapping — using the current value estimate to update itself:

# TD(0) update: estimate V(s) from experience
# After observing transition (s, a, r, s'):

td_target = r + gamma * V[s_next]   # what we observed
td_error  = td_target - V[s]        # how wrong our estimate was
V[s]     += alpha * td_error        # update towards target

# alpha = learning rate (e.g. 0.01)
# gamma = discount factor (e.g. 0.99)

Q-Learning

Q-learning directly learns the optimal action-value function Q*(s,a) without needing a model of the environment. It's off-policy — it learns from any experience regardless of how actions were chosen:

import numpy as np

# Q-table for discrete state/action spaces
Q = np.zeros((n_states, n_actions))

for episode in range(n_episodes):
    state = env.reset()
    done = False
    while not done:
        # Epsilon-greedy exploration
        if np.random.random() < epsilon:
            action = env.action_space.sample()  # explore
        else:
            action = np.argmax(Q[state])         # exploit

        next_state, reward, done, _ = env.step(action)

        # Q-learning update
        td_target = reward + gamma * np.max(Q[next_state])
        Q[state, action] += alpha * (td_target - Q[state, action])

        state = next_state

Exploration vs Exploitation

Frequently Asked Questions