RL from Zero to Agents

By the end you can

• Explain how reinforcement learning differs from supervised learning.
• Name the parts of the RL loop: agent, environment, state, action and reward.
• Describe how trial and error, and the trade-off between exploring and exploiting, let an agent improve.
• Recognise two ways RL goes wrong: reward hacking and rewards too sparse to find.
• See how a language model fits the same loop, with tokens as actions and a verifier as the reward.

What you'll do

No code today: a map of the whole field, shown with a maze and a self-driving car that learn live in your browser. Episode 1 builds the maze from an empty file.

By the end you can

• Write the agent–environment loop and say what passes between the two at each step.
• Build a gridworld environment in plain Python with Gymnasium's reset and step.
• Tell an episode that terminated from one that was truncated, and say why the difference matters.
• Run a random agent for many episodes and measure its returns against the best possible.
• Explain how changing the reward changes what an agent does, and how it can be gamed.
• Describe Go, a chatbot and a robot arm in the same terms: their states, actions and rewards.

What you'll do

Episode 0's maze from an empty file, in plain Python: a gridworld with Gymnasium's reset and step, the agent–environment loop, and a random agent run for 1,000 episodes. numpy and matplotlib only draw the charts.

By the end you can

• Frame a repeated choice as a multi-armed bandit.
• Explain the trade-off between exploring and exploiting, and what goes wrong with too much of either.
• Implement ε-greedy, UCB and Thompson sampling.
• Measure a strategy by its regret and read a regret curve.
• Explain why a fixed ε gives regret that keeps growing in a straight line, while UCB's and Thompson sampling's grow only logarithmically.

What you'll do

A multi-armed bandit simulator that runs ε-greedy, UCB and Thompson sampling side by side on the same arms, with each strategy's regret plotted live as it pulls.

By the end you can

• Describe a problem as an MDP: states, actions, transition probabilities, rewards and a discount γ.
• Compute a discounted return and explain how γ changes what an agent prefers.
• Read the Bellman equation as: the value here is the reward now plus γ times the value of where you land.
• Run value iteration and read both the values and the greedy policy off a heatmap.
• Explain why value iteration needs a model of the environment, which Episode 4 learns to do without.

What you'll do

Value iteration on the gridworld from Episode 1, drawn as a heatmap that fills in one sweep at a time, so you see the Bellman update at work before we write it down.

By the end you can

• Explain the difference between model-based and model-free learning.
• Write the Q-learning update and say what α, γ and the TD target each do.
• Train a tabular agent on FrozenLake and read its greedy policy off the Q-table.
• Explain why Q-learning can learn the best policy while still exploring with ε-greedy.
• Estimate when a table becomes too big to fill, and why that calls for function approximation.

What you'll do

Tabular Q-learning on FrozenLake, with the Q-table filling in as the agent plays, then a count of how big that table gets once the state space grows.

By the end you can

• Explain function approximation and why it lets an agent generalise to states it hasn't seen.
• Turn the Q-learning update into a loss a neural network can minimise.
• Spot a diverging DQN from its loss and Q-value curves.
• Explain how a replay buffer and a target network each make training more stable.
• Run DQN on CartPole in a Kaggle notebook.

What you'll do

DQN on CartPole, failures included: first a naive network that goes unstable and diverges, then the replay buffer and target network that fix it. The code runs in a Kaggle notebook, not in the browser.

By the end you can

• Explain how a policy that outputs probabilities can be trained directly, and why that suits large or continuous action spaces.
• Derive the policy gradient with the log-probability trick.
• Implement REINFORCE from scratch.
• Explain why REINFORCE's gradient estimates are noisy, and how a baseline reduces their variance without adding bias.

What you'll do

REINFORCE from scratch, then the same agent with a baseline subtracted from the return, with the spread of its gradient estimates plotted so you can see the variance drop.

By the end you can

• Explain the actor-critic split: a policy that acts and a value function that judges.
• Compute advantages with GAE and say how λ trades bias against variance.
• Explain what PPO's clipped objective prevents, and how it relates to TRPO's trust region.
• Read KL divergence and clip fraction curves to tell when updates are too large.
• Train PPO on LunarLander in a Kaggle notebook.

What you'll do

PPO on LunarLander, with the KL divergence and clip fraction tracked live as it trains. The code runs in a Kaggle notebook, not in the browser.

By the end you can

• Map the RL loop onto text generation: state, action, policy and episode.
• Read token log-probabilities from a model and add them up into the log-probability of a whole completion.
• Write a simple reward function and use it to score sampled completions.
• Describe the three stages of RLHF, and what the KL penalty is there to prevent.
• Explain how DPO learns from preference pairs without a reward model or an RL loop.

What you'll do

Load a small open model, read its token log-probabilities as a policy's action probabilities, and score its completions with a simple reward function. The code runs in a Kaggle notebook, not in the browser.

By the end you can

• Explain how GRPO computes advantages from a group of sampled answers, and why that removes the value network.
• Write a verifiable reward for maths that checks the final answer and its format.
• Set up GRPO training for a small Qwen model within the limits of Kaggle's free T4 GPUs.
• Read accuracy and response-length curves while a model trains.
• Summarise what DeepSeek-R1 showed about RL with verifiable rewards.

What you'll do

GRPO on a small Qwen model for maths, with a verifiable reward that checks the final answer, tracking accuracy and response length as it trains. It runs in a Kaggle notebook on free T4 GPUs, not in the browser.

By the end you can

• Describe an agent's task in RL terms: observations, tool calls as actions, the episode and the reward.
• Implement reset and step for a multi-turn tool-use task.
• Isolate the agent in a sandbox that resets to the same starting state every episode.
• Design a verifier that checks the outcome, not the agent's claims about it.
• Collect rollouts and read them to see how an agent succeeds and fails.

What you'll do

A multi-turn tool-use environment, such as fixing a broken config or service in a sandbox, with a reset and step interface and a programmatic verifier that decides whether the fix worked.

By the end you can

• Define specification gaming and reward hacking, with examples from games, robotics and language models.
• State Goodhart's law and connect it to reward model overoptimisation.
• Find reward hacking during training by reading rollouts, not only reward curves.
• Harden a verifier against an exploit without rejecting honest solutions.
• Design verifiers adversarially, assuming the policy will search for every gap.

What you'll do

Train a model on the Episode 10 environment, catch it gaming the verifier, then harden the verifier and train again, comparing what the model does before and after.

By the end you can

• Explain what makes an environment verifiable, and what makes it adversarial.
• Write a threat model for a security task and turn it into a verifier.
• Scale an environment's difficulty into a curriculum a model can climb.
• Recognise common ways models fail security environments, and what each failure says about the environment.
• Name the open problems in building environments for infrastructure and security agents.

What you'll do

A walkthrough of a security environment in the style of PatchTheVuln: its threat model, its verifier, how its difficulty scales, and the ways models fail it.