What Is RL?

Reinforcement learning is learning from consequences. An agentAgentThe learner and decision-maker. It chooses an action, sees what happens, and changes how it acts next time. tries something, gets a score for it, and leans towards whatever scored well. Nobody shows it the right move. This lesson builds the whole idea without code or maths, using a maze and a car that learn in front of you, in your browser, then looks at where it goes wrong and how the same idea now trains language models.

Learning from consequences

Nobody teaches you to ride a bike with equations. You wobble, you fall, the ground gives you a score, and your body adjusts. You never see the physics, only what happens.

Diagram · Learning from consequences

That loop is enough to beat people at their own games. In 2016, DeepMind's AlphaGo played move 37 in its second game against Lee Sedol, one of the best Go players in the world. AlphaGo's own estimate put the chance of a human playing that move at about one in ten thousand, and commentators took it for a mistake. It won the game. AlphaGo had started from human games, then improved by playing against itself and being scored on wins, and the score found a move its human teachers never showed it.

Copying a teacher, or learning from a score

Most machine learning is supervised: you show the right answer for every example (this photo is a cat, this one a dog) and the model learns to copy the teacher. Reinforcement learning has no answer key. The agent acts, and some time later it gets a single number. Nobody tells it which of its actions earned that number; it has to work that out.

Diagram · Supervised learning and reinforcement learning

The loop every RL system runs

Here is the world this course starts in: a maze of 9 by 12 squares. The agentAgentThe learner and decision-maker. It chooses an action, sees what happens, and changes how it acts next time. is the dot, the thing that decides. Everything else is the environmentEnvironmentEverything outside the agent: the world it acts in. It takes the agent's action and answers with the next state and a reward.: the walls, the goal and the lava.

• The stateStateWhat the agent knows about its situation right now, such as its square in a maze or the readings from a car's sensors. is what the agent knows about its situation right now. In the maze, that's just the square it's on.
• An actionActionA choice the agent can make in a state, such as moving up or steering left. For a language model, an action is the next token. is a choice it can make there: up, down, left or right. Walking into a wall leaves it where it is.
• The rewardRewardA number the environment gives back after each action, saying how good the result was. The agent tries to collect as much reward as it can over time. is the number the environment sends back after each action.

Diagram · The reinforcement learning loop

One trip round the loop is a step. A whole attempt, from the start square until the agent reaches the goal, falls in the lava or runs out of time, is an episodeEpisodeOne attempt from start to finish, such as one run through a maze. It ends at the goal, in a crash, or when time runs out.. Episode 1 writes this loop from an empty file.

Rewards say what you want, and only that

The reward is the only way to tell the agent what you want. In the maze, reaching the goal scores +1 and falling in the lava −1, and both end the episode. There's also a small penalty of −0.01 for every step. That penalty sounds like a detail. It isn't.

Gridworld

Loading simulation…

Without the penalty, reaching the goal in 21 steps or in 365 scores exactly the same, so the agent has no reason to hurry, and it doesn't. With a hundredth of a point per step, shorter is better, and it goes straight there. You meant "reach the goal, quickly". Without the penalty you only scored "reach the goal", and that's what you got.

Play with the maze

Time to watch one learn. This is a Q-learning agent, one of the methods the course builds later, learning the maze from rewards alone, right now in your browser. At first it knows nothing and wanders. Every time a move leads somewhere good, its estimate of that move goes up a little, and those estimates spread back from the goal, episode after episode.

Two things show what it has learned so far. The heatmap is how good it thinks each square is, which is its sense of valueValueHow much reward the agent can expect from a state onwards if it keeps following its policy. It lets the agent judge a move by where it leads, not only by what it pays straight away.. The arrows are its best move from each square, its policyPolicyThe agent's strategy: which action it takes in each state. Learning means improving the policy.. Follow the arrows from the start and you get the route it would take. The charts underneath count every episode.

Explore or exploit

Every agent faces the same choice at every step: take the move it already knows is good (exploit), or try something it hasn't, which might be better or worse (explore). ε, the exploration rate, is the share of moves it makes at random.

Gridworld

Loading simulation…

The agent that never explores never finds the door. It's certain of the long way, so it never tries anything else. The one that explores stumbles through the door within a few dozen episodes, and its route drops from 28 moves to 4. Here each step costs 0.05, so the long route is worth about −0.5, while a square it has never tried still looks like 0. Once it has tried the door even once, the door wins.

What can go wrong

That's the magic. Now the honest part: three ways reinforcement learning goes wrong, all of them problems in real systems.

It games the score

Put a coin worth +0.3 two squares from the start of the maze, paid out every time, and train for 3,000 episodes.

Gridworld

Loading simulation…

It never reaches the goal, not once, and its total reward is in the tens of thousands. It steps on and off the coin forever, and by the score it's doing brilliantly. It isn't broken: it did exactly what we scored. This is reward hackingReward hackingWhen an agent finds a way to score well that its designer never intended, such as circling a coin forever instead of reaching the goal. The agent optimises what you score, not what you meant., and it comes back in Episode 11, where we catch a language model doing it.

The score never comes

In a maze of 21 by 37 squares with its only reward at the far corner, an agent that starts out moving at random almost never finds it.

Gridworld

Loading simulation…

An agent that never sees a reward has nothing to learn from. Real tasks are often like this, which is why how you design the reward matters so much. Episode 3 shows how value spreads back from a reward once one is found, and Episode 4 learns FrozenLake, where the only reward is at the goal.

It needs a lot of tries

It learns from experience, and it needs a lot of it. The car in the next section crashes about 300 times in its first 600 attempts. That's fine in a simulator. It's not fine with a real car or a production server, which is why simulated environments matter so much. Episode 5 uses a replay buffer to learn more from each experience.

Watch a car learn to drive

The car learns by the same method as the maze, on a closed track. Here is its loop:

• State: five distance sensors, each rounded into one of four bands, so it can only tell 4⁵ = 1,024 situations apart.
• Actions: steer left, go straight or steer right, 9° at a time.
• Reward: progress along the track, about 100 for a lap. Leaving the track costs 5 and ends the attempt.

Nobody tells it how to drive. Its whole "brain" is a table of 1,024 rows, one for each thing its sensors can report, holding how good each of the three steering choices is.

Your LLM is an agent too

Here's why this matters now. Take a row of squares from the maze, and swap the grid for text: every square becomes a token, a piece of a word.

Diagram · Tokens as actions

The state is the text so far. The action is the next token. The reward is a check on the result: did the answer pass the test, did the code run? A program that checks answers like this is a verifierVerifierA program that checks an answer automatically, such as running unit tests or comparing a maths result, and turns the check into a reward. Verifiers let RL train language models on tasks that have a right answer.. This is how ChatGPT was tuned with human feedback (RLHF), how reasoning models such as OpenAI's o1 and DeepSeek-R1 were trained, and how tool-using AI agents are trained today, in environments like the ones this course ends by building.

Where this goes next

The rest of the course turns all of this into code, one idea at a time. It starts in Episode 1 by building the maze loop from an empty file in Python.

5 questions

Check your understanding

Pick an answer to see whether it's right, and why. Wrong ones are worth reading too.

1. 01What's the main difference between supervised learning and reinforcement learning?

   Reinforcement learning uses neural networks; supervised learning doesn't.Supervised learning is shown the right answers; reinforcement learning only gets a score for what it does.Reinforcement learning needs labelled examples; supervised learning doesn't.

2. 02After the agent takes an action, what does the environment send back?

   Only the new state.The best action to take next.The new state and a reward.

3. 03In the race after the door opened, why did only the exploring agent find the shortcut?

   Because it sometimes tried moves it didn't rate best, so it eventually tried the door.Because it had a better learning algorithm.Because the door only opened for the exploring agent.

4. 04With a coin worth +0.3 near the start, the agent never reached the goal in 3,000 episodes. Why?

   It couldn't learn the maze.Collecting the coin again and again scored more than reaching the goal.The coin blocked the way to the goal.

5. 05When a language model is trained with reinforcement learning, what is the action?

   The prompt it's given.The next token it writes.The verifier's verdict on its answer.