==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 12 . REINFORCEMENT LEARNING FROM ZERO
Five Cats, One New Power Each: From Reflex to Learning
============================================================================================
Open any AI textbook and it hands you a taxonomy: five agent names, five bullet-point
features, a table to memorise. "Simple reflex uses condition-action rules. Model-based has
internal state. Goal-based has goals." Then it drills you on definitions. One thing it never
answers: WHY is memory not enough to plan? Why is planning not enough to rank outcomes? Why
does learning still lose when the mouse is simply faster? Those three whys get answered
here, in one room, with numbers. A cat. A mouse. One corridor. Nothing about the room ever
changes -- only the cat's brain does.
ONE ROOM, HELD FIXED
positions : 0 1 2 3 4
start : C . . . M (C = cat at cell 0, M = mouse at cell 4)
Vision: the cat sees one cell to its left and one to its right, nothing further. Textbooks
call a world like this partially observable -- the cat never knows the full board.
Movement: the mouse moves one cell per round, any direction or none, and the cat cannot
know which. That unpredictability is what textbooks call stochastic.
Order: a move now changes which cells are reachable next round -- the chase is sequential,
not a series of independent snapshots.
Players: the mouse is not furniture; it actively escapes. Two players, so multi-agent.
Four properties, and they hold on every rung below. A bigger brain moves no walls and
calms no mouse.
FIVE BRAINS, ONE LADDER
Each brain below is the brain before it plus exactly ONE new power. And a power counts
only if you can name the failure it kills -- a failure you can stage in the corridor and
watch happen. No stageable failure killed, no rung earned. Here is the whole ladder,
named once so the shape is visible, then climbed slowly:
brain 5 (learning) feedback updates the scores and the world-guess
^ + one power: nothing is frozen any more
brain 4 (scorecard) a happiness number pinned on every outcome
^ + one power: outcomes can be RANKED
brain 3 (sketchpad) a goal, plus moves tried in imagination first
^ + one power: futures are simulated before a paw commits
brain 2 (diary) notes that survive after the eyes go blank
^ + one power: memory
brain 1 (twitch) reacts to what the eyes catch right now
Climb it from the bottom. Each failure forces the next rung into existence.
A BRAIN THAT IS ONLY A WIRING TABLE
Give the cat exactly one tool: a wiring table. Each row is an IF-THEN rule -- if this is
what the eyes catch right now, do this. No record of a moment ago. No guess about a
moment ahead.
eyes catch right now -> action
-------------------------------- -----------
mouse one cell to the right -> move right
mouse one cell to the left -> move left
nothing in sight -> wander
Textbooks call this a simple reflex agent. Call it the twitch cat: pure reaction to the
present instant.
Stage its failure. Cat at cell 2, mouse at cell 3 -- one cell away, visible:
round 1: cat at 2, mouse at 3. |2 - 3| = 1, visible. rule fires: move right.
round 2: cat at 3, mouse at 0. mouse ducked behind the wall while the cat moved.
|3 - 0| = 3, eyes blank. rule fires: wander.
A live mouse sits at cell 0, three cells away, and the chase is over. No row of the
wiring table can say "the mouse WAS here a moment ago" -- there is nowhere to keep a past
fact, so a past fact can never fire a present rule.
That failure names the missing power: memory.
A MOUSE BEHIND A WALL FORCES A DIARY
Same cat, plus one power: a diary -- a written note that survives after the eyes go blank.
diary, after round 1: "mouse was at cell 3"
"mouse last moved right"
Rules may now read the diary as well as the eyes:
eyes + diary say -> action
----------------------------------------- --------------------------
can't see mouse + diary says "went right" -> keep moving right
diary says "sat in this cell 3 rounds" -> turn around, stop looping
Textbooks call this a model-based reflex agent; the diary is the "internal state". It
kills the sealed-box failure above: eyes blank, diary full, the cat keeps chasing the
ghost of the mouse. It kills blind loops too: the diary notices the repeated cell.
Now stage the failure this rung cannot beat -- a mouse that fakes:
round 1: cat at 1, mouse at 2. visible. diary writes "right". cat moves right.
round 2: cat at 2, mouse at 4. mouse moved right again. diary: "right".
cat moves right.
round 3: cat at 3, mouse at 0. mouse JUKED -- bolted left, now behind the cat.
eyes blank. diary still says "right". cat charges right, into empty
corridor, while the mouse flees the other way.
Everything in that diary was true when written. "Mouse went right" was a fact -- three
rounds ago. A diary records what happened; it cannot play out what will happen next.
Records are not rehearsals.
So this failure names the next power: imagination.
A JUKE FORCES A SKETCHPAD
Same cat, diary kept, plus one power: a GOAL ("catch the mouse") and a sketchpad -- moves
tried in imagination before the paws commit. On the sketchpad the cat plays out short
chains -- "if I go here, the mouse is probably there, then..." -- and picks a chain that
ends in a catch.
sketchpad, round 3 of the juke:
chain A: keep chasing right -> mouse escapes left -> no catch
chain B: cut back to cell 0 -> mouse runs into me -> CATCH
Textbooks call this a goal-based agent, and the trying-in-imagination is planning. It
kills the juke: instead of chasing the tail, the cat simulates the cut-off and intercepts.
It kills blind loops without even needing the diary rule: a simulated revisit shows up as
a dead branch.
Now stage what a sketchpad cannot do. It asks one question of every chain -- "does this
end in a catch?" -- and that question has exactly two answers:
plan A: right, right, cut the corner -> catch in 5 moves, no danger.
sketchpad reads: CATCH (yes)
plan B: right, through the bear trap -> catch in 40 moves, one mangled paw.
sketchpad reads: CATCH (yes)
Two yeses. Nothing to compare. A yes-or-no label cannot rank one yes above another --
"5 moves and safe" and "40 moves and a trap" read as the same word. Preferring one means
pinning a NUMBER on each outcome, and the sketchpad has none.
So two identical yeses force the next power: a score.
TWO YESES FORCE A SCORECARD
Same cat, plus one power: a happiness number pinned on every outcome. Textbooks call the
pinned number a utility.
outcome score
------------------- -----
catch the mouse +100
one wasted round -1
a bear trap -50
a dead end -5
Now plans stop being yes/no and become arithmetic:
plan A: catch in 5 moves, no trap -> 100 + 5 x (-1) + 0 = +95 <- winner
plan B: catch in 40 moves, one trap -> 100 + 40 x (-1) + (-50) = +10
plan C: catch in 6 moves, no trap -> 100 + 6 x (-1) + 0 = +94
Ranking works; the two-yeses failure is dead. But the mouse is unpredictable -- outcomes
arrive with chances, not certainties. So score a choice by chance-weighted arithmetic:
each outcome's score times how likely it is, added. (A chance is a number between 0 and
1; the chances of all outcomes add to 1.)
chase right: 80% catch (+100), 20% miss (-5)
0.80 x 100 + 0.20 x (-5) = 80.0 - 1.0 = 79.0
cut across: 60% catch (+100), 40% miss (-5)
0.60 x 100 + 0.40 x (-5) = 60.0 - 2.0 = 58.0
79.0 beats 58.0: chase right. Textbooks call this maximising expected utility.
Take the same two options, but a miss while cutting across now costs -40 (the cat ends
up cornered), not -5. Rework the second line on your slate.
Check: 0.60 x 100 + 0.40 x (-40) = 60.0 - 16.0 = 44.0. Chase right, by more.
Two failures survive the scorecard. First: who wrote those numbers? A hand did, at build
time. If the hand guessed "mouse goes right 80% of the time" and the truth is 20%, every
expected score is careful arithmetic on a lie -- and nothing inside the cat can ever
correct it. Second: with many branching futures, the chance-weighted sums explode; the
cat freezes mid-arithmetic while the mouse leaves. (Textbooks: computationally
intractable.)
Frozen, and possibly wrong. That names the last power: updating.
A FROZEN WRONG GUESS FORCES LEARNING
Same cat, plus one power: after each chase a feedback number arrives -- caught the mouse
(reward), wasted rounds or hit a wall (penalty) -- and the cat uses it to nudge its own
numbers. Scores and world-guess are no longer frozen.
Watch a wrong guess correct itself. Built-in belief: "mouse goes right" with confidence
0.70. Truth: this mouse goes left. After each chase, nudge the belief 20% of the way
toward what actually happened (went left counts as 0, went right as 1):
chase 1: mouse left. belief = 0.70 + 0.20 x (0 - 0.70) = 0.70 - 0.14 = 0.56
chase 2: mouse left. belief = 0.56 + 0.20 x (0 - 0.56) = 0.56 - 0.112 = 0.448
chase 3: mouse left. belief = 0.448 + 0.20 x (0 - 0.448) = 0.448 - 0.0896 = 0.3584
From 0.70 to 0.36 in three chases, sliding toward the truth near 0. Given enough chases
the hand-coded lie is gone. Slow arithmetic melts too: the learning cat caches shortcuts
-- "in this exact spot, cut left" -- and stops re-deriving them every round.
Take chase 4, mouse left again. Nudge the 0.3584 once more, same rule.
Check: 0.3584 + 0.20 x (0 - 0.3584) = 0.3584 - 0.07168 = 0.28672.
And what learning cannot fix, staged honestly: the first chases are flailing -- the
numbers start miles from the truth and only feedback moves them (textbooks: the cold-
start problem). And no amount of updating rewrites physics: a mouse faster than the
cat's top speed stays uncaught at every belief value.
SAY IT BACK
Cover everything above. Fill the two right columns from the left two, out loud:
brain new power kills then breaks at
--------- -------------- ------------------------- -------------------------
twitch wiring table (the floor) mouse hides; blind loops
diary memory hidden mouse, loops juke (stale direction)
sketchpad goal + planning juke; intercepts cannot rank two yeses
scorecard utility number ranking, risk, odds frozen guess; slow math
learning updatable dials frozen guess, slow math cold start; raw speed
A power earns its rung only by the failure it removes, and removing one failure uncovers
the next -- a ladder, not a bag of personalities. And on every rung the room stayed put:
vision still one cell, mouse still unpredictable, moves still change the future, two
players still in the game. Brains changed. Walls did not.
Pencil work done. Below, the same five brains in Python -- every round hard-coded, no
loops, so each rung's win and its wall sit in plain assignments you can check against
the traces above.
# ---------------------------------------------------------------
# room: corridor cells 0..4. vision range 1: visible means
# |cat - mouse| <= 1. every round hard-coded.
# ---------------------------------------------------------------
# =================================================================
# brain 1 -- twitch cat (wiring table only)
# =================================================================
print("=== twitch cat ===")
# round 1: cat at 2, mouse at 3
cat = 2
mouse = 3
sees = abs(cat - mouse) <= 1 # |2 - 3| = 1 <= 1 -> True
print("round 1: sees =", sees) # True -> rule fires: move right
cat = cat + 1 # cat now at 3
# round 2: mouse ducked behind the wall to cell 0
mouse = 0
sees = abs(cat - mouse) <= 1 # |3 - 0| = 3 <= 1 -> False
print("round 2: sees =", sees) # False -> rule fires: wander
print("mouse alive at 0; chase abandoned. no memory.")
# =================================================================
# brain 2 -- diary cat (+ memory)
# =================================================================
print("=== diary cat ===")
# round 1: cat at 1, mouse at 2. visible -> diary written.
cat = 1
mouse = 2
sees = abs(cat - mouse) <= 1 # |1 - 2| = 1 -> True
diary = "right" # mouse is to the right: 2 > 1
print("round 1: sees =", sees, " diary =", diary) # True, right -> move right
cat = cat + 1 # cat now at 2
# round 2: mouse moved right again, to 4. eyes blank; diary fires.
mouse = 4
sees = abs(cat - mouse) <= 1 # |2 - 4| = 2 -> False
print("round 2: sees =", sees, " diary =", diary) # False, right -> still chasing!
cat = cat + 1 # cat now at 3
# round 3: mouse JUKED to 0. diary is stale; cat runs the wrong way.
mouse = 0
sees = abs(cat - mouse) <= 1 # |3 - 0| = 3 -> False
print("round 3: sees =", sees, " diary =", diary) # False, right -> WRONG WAY
cat = cat + 1 # cat charges to 4; mouse flees from 0
print("diary held the old direction; records are not rehearsals.")
# =================================================================
# brain 3 -- sketchpad cat (+ goal + planning)
# =================================================================
print("=== sketchpad cat ===")
# cat at 2; mouse predicted at 4. two chains tried in imagination.
cat = 2
mouse_predicted = 4
land_A = cat + 1 + 1 # chain A: right, right -> 2 + 1 + 1 = 4
land_B = cat - 1 - 1 # chain B: left, left -> 2 - 1 - 1 = 0
catch_A = (land_A == mouse_predicted) # 4 == 4 -> True
catch_B = (land_B == mouse_predicted) # 0 == 4 -> False
print("chain A lands", land_A, "catch =", catch_A) # 4, True
print("chain B lands", land_B, "catch =", catch_B) # 0, False
# sketchpad picks chain A: yes beats no.
# two-yeses failure: both plans catch; sketchpad reads yes, yes. stuck.
plan_A_catches = True # 5 moves, no trap
plan_B_catches = True # 40 moves, one bear trap
print("plan A:", plan_A_catches, " plan B:", plan_B_catches, " -> cannot choose")
# =================================================================
# brain 4 -- scorecard cat (+ utility)
# =================================================================
print("=== scorecard cat ===")
score_catch = 100
score_trap = -50
score_per_move = -1
score_A = score_catch + 5 * score_per_move + 0 # 100 - 5 + 0 = 95
score_B = score_catch + 40 * score_per_move + score_trap # 100 - 40 - 50 = 10
score_C = score_catch + 6 * score_per_move + 0 # 100 - 6 + 0 = 94
print("A =", score_A, " B =", score_B, " C =", score_C) # 95, 10, 94 -> A wins
# under uncertainty: chance-weighted scores
expected_chase = 0.80 * 100 + 0.20 * (-5) # 80.0 - 1.0 = 79.0
expected_cut = 0.60 * 100 + 0.40 * (-5) # 60.0 - 2.0 = 58.0
print("chase right:", expected_chase) # 79.0
print("cut across :", expected_cut) # 58.0 -> chase right wins
# =================================================================
# brain 5 -- learning cat (+ updatable numbers)
# =================================================================
print("=== learning cat ===")
belief = 0.70 # built-in guess: mouse goes right 70%
# chase 1: mouse went LEFT (left counts as 0)
belief = belief + 0.20 * (0 - belief) # 0.70 + 0.20 x (-0.70) = 0.56
print("after chase 1:", belief) # 0.56
# chase 2: mouse went LEFT again
belief = belief + 0.20 * (0 - belief) # 0.56 + 0.20 x (-0.56) = 0.448
print("after chase 2:", belief) # 0.448
# chase 3: mouse went LEFT a third time
belief = belief + 0.20 * (0 - belief) # 0.448 + 0.20 x (-0.448) = 0.3584
print("after chase 3:", belief) # 0.3584
print("started 0.70; three chases later 0.36 and falling toward the truth.")
print("cold start: those first chases were spent flailing, not catching.")
----------------------------------------------------------------------------------------------
<- prev: (this is the door to Chapter 12)
-> next: Chapter 12, Part 1: Bandits and Exploration
----------------------------------------------------------------------------------------------
home . archive . source on GitHub
==============================================================================================