Glossary

Every term the course uses, in plain English, A to Z. Each one says which episodes teach it; the ones still to come are listed too, so you can look ahead.

52 terms

A

Action: A choice the agent can make in a state, such as moving up or steering left. For a language model, an action is the next token.
Taught in Ep 00, What Is RL? · Ep 01, The Loop That Runs Every RL System (coming soon) · Ep 08, Your LLM Is a Policy (coming soon)
Actor-critic: A method with two parts: an actor, the policy, that chooses actions, and a critic, a value function, that scores them. The actor learns from the critic's estimates instead of raw returns.
Taught in Ep 07, PPO: The Algorithm Behind Modern RL (coming soon)
Advantage: How much better an action turned out than the policy's average from that state: roughly, the return that followed minus the state's estimated value. Positive advantages make an action more likely, negative ones less.
Taught in Ep 07, PPO: The Algorithm Behind Modern RL (coming soon)
Adversarial environment: An environment built on the assumption that the agent will probe it for weaknesses, or one that pits the agent against an opponent, so it stays a fair test as the agent gets stronger.
Taught in Ep 12, Adversarial Environments: Where RL Is Going (coming soon)
Adversarially robust verifier: A verifier built on the assumption that the policy will look for its weaknesses: it checks the real outcome, resists shortcuts, and is tested against known exploits before training.
Taught in Ep 11, Your Model Will Cheat: Reward Hacking (coming soon)
Agent: The learner and decision-maker. It chooses an action, sees what happens, and changes how it acts next time.
Taught in Ep 00, What Is RL? · Ep 01, The Loop That Runs Every RL System (coming soon)

B

Baseline: A value subtracted from the return before it scales the gradient, usually an estimate of the state's value. The gradient points the same way on average, with much less noise.
Taught in Ep 06, Policy Gradients: Learning the Policy Directly (coming soon)
Bellman equation: The rule that a state's value equals the reward expected next plus γ times the value of the state the agent expects to land in. Value iteration applies it to every state, over and over, until the values stop changing.
Taught in Ep 03, How Good Is This State? Values and the Bellman Equation (coming soon)

C

Clipping: PPO's limit on each update. Once an action's probability ratio to the old policy leaves the range 1 − ε to 1 + ε (ε is often 0.2), the objective stops rewarding further change.
Taught in Ep 07, PPO: The Algorithm Behind Modern RL (coming soon)
Curriculum: An ordering of tasks from easy to hard, so the model earns enough reward early on to learn from and is kept at the edge of what it can do.
Taught in Ep 12, Adversarial Environments: Where RL Is Going (coming soon)

D

Deadly triad: Sutton and Barto's name for the three ingredients that together can make value estimates diverge: function approximation, bootstrapping from your own estimates, and learning off-policy.
Taught in Ep 05, Deep Q-Networks: When Tables Aren't Enough (coming soon)
Discount factor (γ): A number between 0 and 1 that shrinks future rewards: a reward one step away counts γ times as much as one now, two steps away γ² times. A γ near 1 makes the agent far-sighted.
Taught in Ep 03, How Good Is This State? Values and the Bellman Equation (coming soon)

E

Environment: Everything outside the agent: the world it acts in. It takes the agent's action and answers with the next state and a reward.
Taught in Ep 00, What Is RL? · Ep 01, The Loop That Runs Every RL System (coming soon) · Ep 10, Building an RL Environment for an AI Agent (coming soon)
Episode: One attempt from start to finish, such as one run through a maze. It ends at the goal, in a crash, or when time runs out.
Taught in Ep 00, What Is RL? · Ep 01, The Loop That Runs Every RL System (coming soon) · Ep 10, Building an RL Environment for an AI Agent (coming soon)
Exploration: Trying actions that don't look best yet, to find out whether something better exists. Its opposite, exploitation, uses what the agent already knows. An ε-greedy agent explores with probability ε.
Taught in Ep 00, What Is RL? · Ep 02, Explore or Exploit? The Bandit Problem (coming soon) · Ep 04, Q-Learning: Learning Without a Map (coming soon)

F

Function approximation: Estimating values with a model that has a fixed number of parameters, such as a neural network, instead of storing one number per state. States that look alike share what is learnt.
Taught in Ep 05, Deep Q-Networks: When Tables Aren't Enough (coming soon)

G

Generalised advantage estimation (GAE): A way to estimate advantages by blending one-step and many-step estimates with a weight λ. A λ near 0 gives low variance but more bias; a λ near 1 gives the reverse.
Taught in Ep 07, PPO: The Algorithm Behind Modern RL (coming soon)
Goodhart's law: When a measure becomes a target, it ceases to be a good measure. In RL, the harder you optimise a proxy reward, the further it can drift from what you actually wanted.
Taught in Ep 11, Your Model Will Cheat: Reward Hacking (coming soon)
Group-relative advantage: An answer's reward minus the mean reward of its group, divided by the group's standard deviation. Answers better than the group are made more likely; worse ones less.
Taught in Ep 09, GRPO on a Free GPU (coming soon)
GRPO: Group relative policy optimisation: a PPO-style method that samples several answers to each prompt and uses each answer's reward relative to the group as its advantage, so it needs no learnt value function.
Taught in Ep 09, GRPO on a Free GPU (coming soon)

K

KL divergence: A measure of how different two probability distributions are. In PPO it tracks how far the new policy has moved from the old one; in RLHF it keeps the tuned model close to where it started.
Taught in Ep 07, PPO: The Algorithm Behind Modern RL (coming soon)

L

Log-probability: The logarithm of the probability a model gave a token. Adding them up gives the log-probability of a whole completion, the quantity policy gradient methods push up or down.
Taught in Ep 08, Your LLM Is a Policy (coming soon)
Log-probability trick: A rewrite of the gradient of the expected return as an average, over sampled actions, of the return times the gradient of each action's log-probability. It lets us estimate the gradient from episodes we've played.
Taught in Ep 06, Policy Gradients: Learning the Policy Directly (coming soon)

M

Markov decision process (MDP): The standard model of an RL problem: states, actions, the probability of each next state, a reward and a discount γ. Markov means the next state depends only on the current state and action, not on how the agent got there.
Taught in Ep 03, How Good Is This State? Values and the Bellman Equation (coming soon)
Model-free: Learning from experience without knowing how the environment works: no transition probabilities, only the states, actions and rewards that actually happened.
Taught in Ep 04, Q-Learning: Learning Without a Map (coming soon)
Multi-armed bandit: A problem with one situation and several options, each paying out at random from its own unknown distribution. The agent has to find the best option while it is still earning from all of them.
Taught in Ep 02, Explore or Exploit? The Bandit Problem (coming soon)

P

Policy: The agent's strategy: which action it takes in each state. Learning means improving the policy.
Taught in Ep 00, What Is RL? · Ep 03, How Good Is This State? Values and the Bellman Equation (coming soon) · Ep 06, Policy Gradients: Learning the Policy Directly (coming soon) · Ep 08, Your LLM Is a Policy (coming soon)

Q

Q-value: The return expected from taking a given action in a given state and acting well afterwards. A Q-table stores one for every pair of state and action.
Taught in Ep 04, Q-Learning: Learning Without a Map (coming soon)

R

Regret: The reward an agent loses by not always picking the best option: what the best arm would have paid on average over all the pulls, minus what the agent's choices paid on average. Lower is better.
Taught in Ep 02, Explore or Exploit? The Bandit Problem (coming soon)
Replay buffer: A store of past transitions: state, action, reward and next state. DQN trains on random batches from it, which breaks up runs of similar, correlated steps and reuses each experience many times.
Taught in Ep 05, Deep Q-Networks: When Tables Aren't Enough (coming soon)
Return: All the reward an agent collects in one episode, added up. It's the score an agent is really trying to raise. The discounted return counts later rewards for a little less.
Taught in Ep 01, The Loop That Runs Every RL System (coming soon)
Reward: A number the environment gives back after each action, saying how good the result was. The agent tries to collect as much reward as it can over time.
Taught in Ep 00, What Is RL? · Ep 01, The Loop That Runs Every RL System (coming soon)
Reward hacking: When an agent finds a way to score well that its designer never intended, such as circling a coin forever instead of reaching the goal. The agent optimises what you score, not what you meant.
Taught in Ep 00, What Is RL? · Ep 11, Your Model Will Cheat: Reward Hacking (coming soon) · Ep 12, Adversarial Environments: Where RL Is Going (coming soon)
Reward model: A network trained to predict which of two answers a person would prefer. Its score stands in for human judgement during RL.
Taught in Ep 08, Your LLM Is a Policy (coming soon)
RLHF: Reinforcement learning from human feedback: fine-tune a model on examples, train a reward model on people's preferences between answers, then optimise the model against that reward with RL such as PPO.
Taught in Ep 08, Your LLM Is a Policy (coming soon)
Rollout: One run of the policy through the environment, from reset to the end of the episode, recorded step by step. Training learns from batches of rollouts.
Taught in Ep 10, Building an RL Environment for an AI Agent (coming soon)

S

Sandbox: An isolated environment, such as a container, where an agent can run commands and change files without touching anything real, and which can be reset to the same starting state for every episode.
Taught in Ep 10, Building an RL Environment for an AI Agent (coming soon)
Specification gaming: Meeting the letter of an objective while missing its point, such as passing a test by special-casing it instead of fixing the code. Reward hacking is specification gaming of a reward.
Taught in Ep 11, Your Model Will Cheat: Reward Hacking (coming soon)
State: What the agent knows about its situation right now, such as its square in a maze or the readings from a car's sensors.
Taught in Ep 00, What Is RL? · Ep 01, The Loop That Runs Every RL System (coming soon)
Stochastic policy: A policy that gives each action a probability instead of always picking one. Sampling from it explores on its own, and gradient descent can change its probabilities smoothly.
Taught in Ep 06, Policy Gradients: Learning the Policy Directly (coming soon)
Supervised fine-tuning (SFT): Training a pretrained model on example prompts with good answers, so it already follows instructions before RL starts.
Taught in Ep 08, Your LLM Is a Policy (coming soon)

T

Target network: A copy of the Q-network, refreshed only every so often, that computes the targets the main network learns towards. Holding the targets still for a while stops the network chasing its own changing estimates.
Taught in Ep 05, Deep Q-Networks: When Tables Aren't Enough (coming soon)
Temporal-difference (TD) update: Moving an estimate a step of size α towards a better-informed target: the reward just received plus γ times the estimate for the next state. It learns after every step, without waiting for the episode to end.
Taught in Ep 04, Q-Learning: Learning Without a Map (coming soon)
Termination: An episode ending because the world says so: the agent reached the goal, fell in the lava, crashed. After it there is no more reward to come.
Taught in Ep 01, The Loop That Runs Every RL System (coming soon)
Thompson sampling: A strategy that keeps a belief about each arm's payout, draws one plausible value per arm from those beliefs, and pulls the arm whose draw is highest.
Taught in Ep 02, Explore or Exploit? The Bandit Problem (coming soon)
Threat model: A statement of what an attacker wants, what they can do and what the defender must protect. In an environment it fixes what counts as success and which shortcuts must fail.
Taught in Ep 12, Adversarial Environments: Where RL Is Going (coming soon)
Tool use: An agent acting by calling functions, such as running a shell command or reading a file, and getting their output back as its next observation.
Taught in Ep 10, Building an RL Environment for an AI Agent (coming soon)
Truncation: An episode being stopped from outside, usually because it ran out of time. The world could have gone on, so the future still counts. Gymnasium's step() reports it apart from termination.
Taught in Ep 01, The Loop That Runs Every RL System (coming soon)

U

Upper confidence bound (UCB): A strategy that picks the arm with the highest optimistic estimate: its average reward so far plus a bonus that shrinks the more often the arm has been tried.
Taught in Ep 02, Explore or Exploit? The Bandit Problem (coming soon)

V

Value: How much reward the agent can expect from a state onwards if it keeps following its policy. It lets the agent judge a move by where it leads, not only by what it pays straight away.
Taught in Ep 00, What Is RL? · Ep 03, How Good Is This State? Values and the Bellman Equation (coming soon)
Verifiable reward: A reward computed by checking the answer with a program, such as comparing a maths result with the known solution, rather than by a learnt model's opinion.
Taught in Ep 09, GRPO on a Free GPU (coming soon)
Verifier: A program that checks an answer automatically, such as running unit tests or comparing a maths result, and turns the check into a reward. Verifiers let RL train language models on tasks that have a right answer.
Taught in Ep 00, What Is RL? · Ep 09, GRPO on a Free GPU (coming soon) · Ep 10, Building an RL Environment for an AI Agent (coming soon) · Ep 11, Your Model Will Cheat: Reward Hacking (coming soon) · Ep 12, Adversarial Environments: Where RL Is Going (coming soon)