==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 12 . REINFORCEMENT LEARNING . PART 4 OF 5
Finding the Best Plan by Pencil: How Good Arrows Spread From the Runway
============================================================================================
The companion post, Grading a Plan by Pencil, took a
plan you were HANDED -- a fixed arrow on every spot -- and filled in the worth of each spot.
This post is the harder half: you are handed NO plan. The notebook of arrows is blank, and
FINDING the best arrows is the whole job. Every reinforcement-learning post races past this
with "...and then it converges to the optimal policy." Two things they skip: (1) the actual
move that makes a plan better -- peek the handful of nudges, keep the winner -- worked on a
toy with arithmetic; (2) WHY a plan that stops changing is provably the best one, not merely
a plan that got tired. This page shows both, by hand. No magic.
TOY WORLD, REAL RECIPE. Grade a plan, peek every nudge, keep the best, repeat until the
arrows freeze -- exact. Real problems let a move slip -- push North, sometimes drift East --
and then a peek averages over where you MIGHT land, weighted by the odds. Here a push lands
for sure, so that average collapses to a single spot. That collapse is the toy, named again
loudly at the end. Trust the recipe; treat the rest as a sketchpad.
You push a pawn to E4. Ten moves later you win. Was E4 a good move, or did you just
get lucky after it? The instant you played it the reward was 0 -- so its own reward
cannot tell you a thing.
E4 --> ... --> ... --> ... (10 moves) --> WIN (+1)
^_____________ but E4 itself paid 0 at the time it was played
So judge it from the end backward. Hand the winning position its worth, then let
that worth bleed one step back to the position before it, then another step back,
and on down the chain:
WIN: +1
<-- 0.9 <-- 0.81 <-- ... <-- E4 inherits its share
E4's worth becomes the best worth of whatever it can reach next. The future pays
the move that led to it, even a move made ten turns earlier. That backward bleed
is Bellman's idea.
Named here so you are not surprised later:
To better a plan at a spot, do not solve anything. Just PEEK down each nudge -- "if I went
THIS way, what would the spot I land on be worth?" -- and re-aim the arrow at the best
peek. Grade the new plan, peek again, and repeat. In this toy the good arrows happen to
spread outward from the runway one ring per pass, then stop. A plan that survives a full
peek untouched is provably the best.
WORLD AND TERMS (DEFINED HERE, COMPLETE)
A drone hangs in the air over a patch of enemy ground. We want to find the best flight plan.
a SPOT = a place the drone can be: a position (x, y, height).
the GROUND = the lowest spots (height 0). Touching the ground ENDS the flight.
the RUNWAY = the middle ground spot. Touch it -> +5 points. (the prize)
DIRT = any other ground spot. Touch it -> -2 points. (a crash)
the AIR = any spot above the ground. The drone does not stop here; it must move on.
FUEL = sitting in the air for one move costs -0.1 points. (the meter always runs)
a NUDGE = one move: N, S, E, W (sideways), U, D (up, down). It changes one number by 1.
the WALL = the edge of the box. A nudge that would leave the box keeps the drone put.
the SHRINK = a worth one move into the future counts as 0.9 of itself. (the famous "g")
Painted facts: +5, -2, -0.1, 0.9 -- handed to you, fixed.
WORLD, DRAWN (3x3 ground, 3 heights; top view and side view):
TOP VIEW (height 0 = ground): SIDE VIEW (centre column, y=1):
DIRT DIRT DIRT height 2: [ air ] <- drone in the air
DIRT RUNWY DIRT <- runway centre height 1: [ air ]
DIRT DIRT DIRT height 0: [RUNWAY ] = +5 (ground, ends flight)
x: 0 1 2
A nudge D takes the drone one height lower.
Six nudges: N S E W (sideways) + U D (vertical).
SIX NUDGES from spot (x, y, height):
N -> (x, y+1, height) S -> (x, y-1, height)
E -> (x+1, y, height) W -> (x-1, y, height)
U -> (x, y, height+1) D -> (x, y, height-1)
A nudge that leaves the box: the drone stays put. (Wall.)
Two notebooks, the only paper we keep:
the PLAN = a notebook, spot -> one arrow (which nudge to make here). The thing we build.
the WORTH = a notebook, spot -> a number (points collected from here on under the plan).
Scratch paper: rebuilt from nothing each time the plan changes.
GRADE STEP, FOLDED INTO ONE BOX (THE COMPANION POST, COMPRESSED)
Before a plan can be bettered, it must be GRADED -- every spot given a worth under the current
arrows. That is the companion post's entire job; here is the whole of it in one box, so you
need nothing external.
worth(ground) = its painted points (+5 runway / -2 dirt -- no lookup)
worth(air) = -0.1 + 0.9 * worth(the spot the plan's arrow points to)
start every worth at 0 (a deliberate lie), lay the two lines above down over all spots
again and again; the painted ground is a fixed anchor the truth climbs out from, one spot
per pass; stop when a pass moves nothing.
One fact we lean on hard below: grade the column straight above the runway and you get +5 at
the pad, then -0.1 + 0.9*5 = 4.4 one up, then -0.1 + 0.9*4.4 = 3.86 two up. Worth is
biggest right over the pad and shrinks with height above it -- a property of this map, one
prize, one path, no competing rewards. Hold that picture for the worked example.
GRADING ONLY READS ONE ARROW -- IMPROVING MUST SEE ALL SIX
Grading read ONE arrow per spot -- whatever the plan said -- and followed it. To IMPROVE a
plan we must compare the arrow it has against the arrows it could have. So for the first time
we need every nudge a spot offers, and where each one lands:
N : (x, y+1, height) go north U : (x, y, height+1) go up
S : (x, y-1, height) go south D : (x, y, height-1) go down
E : (x+1, y, height) go east
W : (x-1, y, height) go west
The wall rule, worked once: at (0, 0, 1) a nudge W would set x = -1, off the box -- so the
drone stays at (0, 0, 1). A wall nudge lands you back on yourself. A ground spot has no nudge
at all (the flight is over).
IMPROVE MOVE -- PEEK THE SIX, KEEP THE WINNER
Stand on an air spot. You hold a worth for every spot (from the grade). You want this spot's
BEST arrow. Do not solve anything. PEEK each nudge and score it:
score(nudge) = -0.1 + 0.9 * worth(the spot that nudge lands on)
|fuel| |0.9 times a number read from the WORTH notebook|
Six nudges, six scores. Re-aim the arrow at the biggest. That is the entire improve move.
PEEK AT ONE SPOT, DRAWN (standing at spot S, six directions, one score each):
U -> worth(above) * 0.9 - 0.1
^
W -> worth <-- [ S ] --> E -> worth
v
D -> worth(below) * 0.9 - 0.1
(N and S go into/out of the page -- same calculation, different neighbour worth)
best arrow here = the nudge with the largest score
(Textbooks write one peek's score as Q(s, a) and the winner as argmax over a. Same thing.)
Note what does NOT change across the six: the fuel -0.1 is the same for every nudge (it is the
cost of being in the air this move, not of the direction). Only 'worth(where it lands)' moves.
So the peek really asks one thing: which neighbour is worth the most? Aim there.
WHOLE LOOP -- GRADE, PEEK, REPEAT, FREEZE
No worths exist before a plan exists, and no smart plan exists before worths exist -- another
circle. Break it the same way grading broke its circle: start from a lie. Scribble ANY plan.
Garbage is fine; it gets fixed.
LOOP, DRAWN:
START: scribble ANY plan (e.g. "always go Left")
|
v
GRADE: fill every spot's worth under this plan
(start worths at 0, lay the Bellman law down again and again until settled)
|
v
IMPROVE: at every air spot, peek all 6 nudges, re-aim the arrow at the winner
|
v
Any arrow moved?
YES ---> go back to GRADE (plan changed; old worths are now wrong; rebuild them)
NO ---> STOP. the plan is frozen. the frozen plan is the best.
Two layers of repeating live here, and mixing them up is the usual stumble. INSIDE the grade,
many passes settle the worths for ONE plan. OUTSIDE, the grade-then-improve cycle swaps that
plan for a better one. PLAN is the thread that carries across and keeps getting better;
WORTH is scratch paper, rebuilt from 0 every time the plan changes.
WATCH THE GOOD ARROWS SPREAD -- WORKED BY HAND
Seed the dumb plan "always nudge Left" (every air arrow = W). The drone slides into the
west wall and never lands. Grade it: an air spot that keeps bumping a wall lands on itself
forever, so its worth solves worth = -0.1 + 0.9*worth, i.e. 0.1*worth = -0.1, i.e. worth = -1.
So after grading:
WORTH: every air spot = -1 ; runway (1,1,0) = +5 ; every dirt spot = -2
Now IMPROVE the spot just above the runway, (1,1,1). Peek its six nudges on this -1 board:
D -> lands (1,1,0) = +5 : score = -0.1 + 0.9*( +5 ) = 4.4 <- biggest
N -> lands air = -1 : score = -0.1 + 0.9*( -1 ) = -1.0
S -> lands air = -1 : score = -0.1 + 0.9*( -1 ) = -1.0
E -> lands air = -1 : score = -0.1 + 0.9*( -1 ) = -1.0
W -> lands air = -1 : score = -0.1 + 0.9*( -1 ) = -1.0
U -> lands air = -1 : score = -0.1 + 0.9*( -1 ) = -1.0
The winner is D, by a mile. The arrow at (1,1,1) flips W -> D. An arrow moved, so we keep
going. GRADE again: now (1,1,1) drops onto the runway, so its worth becomes -0.1 + 0.9*5 = 4.4.
Now IMPROVE the spot one nudge west of it, (0,1,1). Peek on the freshly graded board:
E -> lands (1,1,1) = 4.4 : score = -0.1 + 0.9*( 4.4 ) = 3.86 <- biggest now
D -> lands (0,1,0) = -2 : score = -0.1 + 0.9*( -2 ) = -1.9
others -> air = -1 : score = -0.1 + 0.9*( -1 ) = -1.0
The winner is E. The arrow at (0,1,1) flips toward (1,1,1). Another arrow moved -- keep going.
CHECK: spot (0,1,1) also has nudges N and S (not shown above). N lands (0,2,1), S lands
(0,0,1) -- both still worth -1 on this board (not yet improved). Score for N = score for S =
-0.1 + 0.9*(-1) = -1.0. E (3.86) beats all six. The skipped directions confirm the winner.
ARROWS SPREAD (top view, height 1 layer, in this toy):
PASS 0 -- dumb seed "always Left" (W) everywhere:
W W W
W W W all arrows wrong; worth of every air spot = -1
W W W
PASS 1 -- improve; only the spot directly above the runway (centre) finds D is better:
W W W
W D W centre arrow flips W -> D (toward runway below it)
W W W
(1 arrow moved)
PASS 2 -- grade again; the ring around the centre now sees E or N or S toward D is best:
W D W
E D W neighbours of the centre flip toward it
W D W
(4 arrows moved)
PASS 3 -- outer ring follows:
E D W or S
E D W
E D W or N
(more arrows flip; eventually all point toward the runway column, then down)
See the shape in THIS toy. Pass one fixed the spot touching the runway. Pass two fixed the
spot touching THAT. In this environment -- one prize, uniform fuel, no slipping -- the good
arrows happen to spread outward from the runway one ring per pass, each ring pointing the
short way to the ring already fixed. This is a property of the map, not a law of policy
iteration: in a more complex world multiple distant spots can flip at once, and improvements
need not move in neat rings. When a whole pass re-aims nothing -- the frozen condition --
that is the universal signal.
WHY A FROZEN PLAN IS THE BEST (DERIVED, NOT DECLARED)
"It stopped changing" sounds like it merely ran out of energy. It is stronger than that.
Three facts, in order, force the frozen plan to be optimal:
(a) IMPROVE never makes a plan worse. At any spot, the arrow the plan ALREADY had is one
of the nudges the peek looks at. The winner scores at least as high as the incumbent.
So the new plan is at least as good as the old one at every spot. Never a step down.
(b) Each improve step either finds a strictly better plan OR finds none (frozen). It cannot
produce a plan that is merely "equal but different" and keep going -- any arrow that
moved means at least one spot is now strictly better served, so the new plan is
genuinely distinct from the old. There are only finitely many deterministic plans
(one arrow per air spot, finite choices each). A sequence that is strictly improving
over a finite set cannot cycle and cannot run forever. It must stop.
(c) When it stops, here is what "stopped" means: at every spot, the arrow already there
beats all the other nudges, judged against worths that came FROM this very plan. A plan
that is greedy with respect to its own value function -- its own best reply -- satisfies
the Bellman optimality conditions. The Policy Improvement Theorem (a real proof, not a
definition) guarantees that such a plan is optimal: no other plan can score higher. The
plan is not optimal because we declare it so; it is optimal because satisfying those
conditions is what optimality means, and the theorem proves the two are the same thing.
So freezing is not exhaustion -- it is a fixed point. The plan answers its own question and
wins.
CHECK: if M's arrow flipped BACK from 'down' to 'up' after freezing, the grade would give M
a worth of -1 (wall loop: -0.1 + 0.9*(-1) = -1). Peeking again with that worth:
score_down = -0.1 + 0.9*5 = 4.4, score_up = -0.1 + 0.9*(-1) = -1.0. Down wins immediately.
'Up' cannot survive its own peek. The only survivor is the plan where every arrow wins its
own peek -- the frozen one.
(Textbook names, pinned once: a spot is a state, a plan is a policy, a nudge is an action,
the worth is the value function V. Grading a fixed plan is policy evaluation; peeking every
nudge to re-aim arrows is policy improvement; the whole grade-improve-repeat cycle is policy
iteration. The frozen plan that wins every peek under its own worth table is the optimal
policy, written pi*. Shrink 0.9 is the discount factor gamma. One nudge's score is the
action-value Q(s,a); the nudge that wins the peek is argmax_a Q(s,a).)
In this toy -- one prize, uniform fuel cost -- worth is biggest right over the runway and
shrinks with distance, so the winning peek at any spot is the nudge that cuts distance to
the pad-column. In a world with prizes scattered elsewhere or stochastic slipping, the worth
table and the best plan can look completely different. Here the math hands back exactly the
plan a human would draw for this map.
PYTHON: SIX PEEKS BY HAND, THEN THREE PASSES UNROLLED
Three blocks. Cases 1 and 2 are fully explicit -- every score written out, winner visible.
Case 3 unrolls three grade-improve passes without any loop, so you can trace each line
against the walk above.
# -----------------------------------------------------------------------
# THE PAINTED FACTS (handed to us, fixed)
# runway -> +5 dirt -> -2 fuel in the air -> -0.1 shrink -> 0.9
# -----------------------------------------------------------------------
RUNWAY, DIRT, FUEL, SHRINK = 5.0, -2.0, -0.1, 0.9
# =======================================================================
# CASE 1: ONE IMPROVE, BY HAND -- peek all six nudges at (1,1,1).
# Board: dumb 'always-Left' plan, freshly graded. Every air spot settled
# to -1 (slides into the wall, lands on itself: -0.1 + 0.9*x = x -> x = -1).
# All six scores written out; the winner is not hidden inside a max().
# =======================================================================
print("=== CASE 1: improve (1,1,1) -- peek the six by hand ===")
worth_runway = RUNWAY # D from (1,1,1) lands the runway (1,1,0) = +5
worth_air = -1.0 # every other nudge lands an air spot, all worth -1 on this board
score_D = FUEL + SHRINK * worth_runway # -0.1 + 0.9*( 5) = 4.4 <- lands the prize
score_N = FUEL + SHRINK * worth_air # -0.1 + 0.9*(-1) = -1.0
score_S = FUEL + SHRINK * worth_air # -0.1 + 0.9*(-1) = -1.0
score_E = FUEL + SHRINK * worth_air # -0.1 + 0.9*(-1) = -1.0
score_W = FUEL + SHRINK * worth_air # -0.1 + 0.9*(-1) = -1.0 (the OLD arrow)
score_U = FUEL + SHRINK * worth_air # -0.1 + 0.9*(-1) = -1.0
print(f" D={score_D:.2f} N={score_N:.2f} S={score_S:.2f} "
f"E={score_E:.2f} W={score_W:.2f} U={score_U:.2f}")
print(f" winner = D ({score_D:.2f}). arrow flips W -> D. (an arrow moved -> keep going)")
# D=4.40 N=-1.00 S=-1.00 E=-1.00 W=-1.00 U=-1.00
# =======================================================================
# CASE 2: THE SPREAD -- regrade, then improve the next ring out, (0,1,1).
# After (1,1,1) points D, regrading gives worth(1,1,1) = -0.1 + 0.9*5 = 4.4.
# Now the spot west of it can find a good arrow: aim at 4.4.
# =======================================================================
print("\n=== CASE 2: the good arrow spreads one ring out, to (0,1,1) ===")
worth_111 = FUEL + SHRINK * RUNWAY # regraded: -0.1 + 0.9*5 = 4.4
worth_010 = DIRT # (0,1,0) is dirt = -2
score_E2 = FUEL + SHRINK * worth_111 # -0.1 + 0.9*(4.4) = 3.86 <- toward the ring
score_D2 = FUEL + SHRINK * worth_010 # -0.1 + 0.9*(-2) = -1.9 (straight down = crash)
score_other2 = FUEL + SHRINK * (-1.0) # -0.1 + 0.9*(-1) = -1.0 (still-bad air)
print(f" E={score_E2:.2f} D={score_D2:.2f} others={score_other2:.2f}")
print(f" winner = E ({score_E2:.2f}). (0,1,1) now points at (1,1,1). the good plan grew a ring.")
# E=3.86 D=-1.90 others=-1.00
# =======================================================================
# CASE 3: THREE PASSES, FULLY UNROLLED -- no loop, no def, no hidden state.
# World: the runway column only. Three spots:
# G = runway (ground, painted +5, game ends here)
# M = one height up (air; nudge down lands G, nudge up lands T)
# T = two heights up (air; nudge down lands M, nudge up hits ceiling -> stays T)
# Dumb seed plan: M->up, T->up.
# =======================================================================
print("\n=== CASE 3: three grade-improve passes, fully unrolled ===")
# -------- PASS 1 --------
# Grade plan {M->up, T->up}.
# T follows 'up' -> hits ceiling -> lands T again.
# Steady state: worth_T = FUEL + SHRINK*worth_T
# => (1-SHRINK)*worth_T = FUEL => worth_T = FUEL/(1-SHRINK) = -0.1/0.1 = -1.0
# M follows 'up' -> T (worth=-1): worth_M = FUEL + SHRINK*(-1) = -1.0
worth_G = RUNWAY # 5.0 (painted, never changes)
worth_T = FUEL / (1.0 - SHRINK) # -0.1 / 0.1 = -1.0 (wall loop)
worth_M = FUEL + SHRINK * worth_T # -0.1 + 0.9*(-1) = -1.0
print(f"pass 1 grade: G={worth_G:.1f} M={worth_M:.2f} T={worth_T:.2f}")
# Improve M: peek down (lands G=5) and up (lands T=-1), using graded worths.
score_M_dn = FUEL + SHRINK * worth_G # -0.1 + 0.9*5 = 4.40 <- winner
score_M_up = FUEL + SHRINK * worth_T # -0.1 + 0.9*(-1) = -1.00
plan_M = 'down' # flips from 'up'; an arrow moved
print(f"pass 1 improve M: dn={score_M_dn:.2f} up={score_M_up:.2f} -> {plan_M} (arrow moved)")
# Improve T: peek down (lands M=-1 same board) and up (wall=T=-1).
score_T_dn = FUEL + SHRINK * worth_M # -0.1 + 0.9*(-1) = -1.00 (M still worth -1 this board)
score_T_up = FUEL + SHRINK * worth_T # -0.1 + 0.9*(-1) = -1.00 (tie)
plan_T = 'up' # tie -- no change
print(f"pass 1 improve T: dn={score_T_dn:.2f} up={score_T_up:.2f} -> {plan_T} (tie, no change)")
print("pass 1: M moved -> keep going")
# -------- PASS 2 --------
# Grade plan {M->down, T->up}.
# M follows 'down' -> G: worth_M = FUEL + SHRINK*5 = 4.4
# T follows 'up' -> ceiling -> T: same wall loop -> worth_T = -1 again
worth_T = FUEL / (1.0 - SHRINK) # -1.0 (T->up still a wall loop)
worth_M = FUEL + SHRINK * RUNWAY # -0.1 + 0.9*5 = 4.40
print(f"\npass 2 grade: G={worth_G:.1f} M={worth_M:.2f} T={worth_T:.2f}")
# Improve M (already down, confirm).
score_M_dn = FUEL + SHRINK * RUNWAY # -0.1 + 0.9*5 = 4.40
score_M_up = FUEL + SHRINK * worth_T # -0.1 + 0.9*(-1) = -1.00
plan_M = 'down' # no change
print(f"pass 2 improve M: dn={score_M_dn:.2f} up={score_M_up:.2f} -> {plan_M} (no change)")
# Improve T: down lands M (now worth 4.4); up is wall -> stays T (worth -1).
score_T_dn = FUEL + SHRINK * worth_M # -0.1 + 0.9*4.4 = 3.86 <- winner
score_T_up = FUEL + SHRINK * worth_T # -0.1 + 0.9*(-1) = -1.00
plan_T = 'down' # flips from 'up'; an arrow moved
print(f"pass 2 improve T: dn={score_T_dn:.2f} up={score_T_up:.2f} -> {plan_T} (arrow moved)")
print("pass 2: T moved -> keep going")
# -------- PASS 3 (confirm freeze) --------
# Grade plan {M->down, T->down}.
# M follows 'down' -> G: worth_M = 4.4
# T follows 'down' -> M (worth 4.4): worth_T = FUEL + SHRINK*4.4 = 3.86
worth_M = FUEL + SHRINK * RUNWAY # 4.40
worth_T = FUEL + SHRINK * worth_M # -0.1 + 0.9*4.4 = 3.86
print(f"\npass 3 grade: G={worth_G:.1f} M={worth_M:.2f} T={worth_T:.2f}")
# Improve M: down=G (score 4.40) vs up=T=3.86 (score 3.374).
score_M_dn = FUEL + SHRINK * RUNWAY # 4.40
score_M_up = FUEL + SHRINK * worth_T # -0.1 + 0.9*3.86 = 3.374
plan_M = 'down' # 4.40 > 3.374, no change
print(f"pass 3 improve M: dn={score_M_dn:.2f} up={score_M_up:.3f} -> {plan_M} (no change)")
# Improve T: down=M (score 3.86) vs up=T(wall, worth 3.86, score 3.374).
score_T_dn = FUEL + SHRINK * worth_M # -0.1 + 0.9*4.4 = 3.86
score_T_up = FUEL + SHRINK * worth_T # -0.1 + 0.9*3.86 = 3.374 (wall: nudge up lands T)
plan_T = 'down' # 3.86 > 3.374, no change
print(f"pass 3 improve T: dn={score_T_dn:.2f} up={score_T_up:.3f} -> {plan_T} (no change)")
print("pass 3: no arrows moved -> FROZEN. best plan: M->down, T->down")
print(f"frozen worths: T={worth_T:.2f} M={worth_M:.2f} G={worth_G:.1f}")
Output of the three blocks:
=== CASE 1: improve (1,1,1) -- peek the six by hand ===
D=4.40 N=-1.00 S=-1.00 E=-1.00 W=-1.00 U=-1.00
winner = D (4.40). arrow flips W -> D. (an arrow moved -> keep going)
=== CASE 2: the good arrow spreads one ring out, to (0,1,1) ===
E=3.86 D=-1.90 others=-1.00
winner = E (3.86). (0,1,1) now points at (1,1,1). the good plan grew a ring.
=== CASE 3: three grade-improve passes, fully unrolled ===
pass 1 grade: G=5.0 M=-1.00 T=-1.00
pass 1 improve M: dn=4.40 up=-1.00 -> down (arrow moved)
pass 1 improve T: dn=-1.00 up=-1.00 -> up (tie, no change)
pass 1: M moved -> keep going
pass 2 grade: G=5.0 M=4.40 T=-1.00
pass 2 improve M: dn=4.40 up=-1.00 -> down (no change)
pass 2 improve T: dn=3.86 up=-1.00 -> down (arrow moved)
pass 2: T moved -> keep going
pass 3 grade: G=5.0 M=4.40 T=3.86
pass 3 improve M: dn=4.40 up=3.374 -> down (no change)
pass 3 improve T: dn=3.86 up=3.374 -> down (no change)
pass 3: no arrows moved -> FROZEN. best plan: M->down, T->down
frozen worths: T=3.86 M=4.40 G=5.0
ONE HONEST FOOTNOTE. Each peek here reads ONE landing spot: '-0.1 + 0.9 * worth(where the
nudge lands)'. Real problems let a nudge slip -- push down, sometimes drift sideways -- so
a peek must average over where you MIGHT land: '-0.1 + 0.9 * sum over landings of
P(landing) * worth(landing)'. Our nudges land for sure, so P = 1 for the one spot and 0
for the rest, and the average collapses to the single term you saw. That is the only thing
the toy hides. Put the weighted average back and every peek, every spread, every freeze on
this page holds.
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 12 -- Reinforcement Learning):
Part 1 -- Five Cats, One New Power Each
Part 2 -- Greedy Shopkeeper, Faulty Machines
Part 3 -- Worth and the Bellman Recipe from Zero
Part 3 -- Grading a Plan by Pencil
Part 4 (this post) -- Finding the Best Plan by Pencil
Part 5 -- No Map in the Arguments: Value Iteration
NEXT CHAPTER: Chapter 13, Part 1: TD(0) Built by Pencil
<- Back to all posts
----------------------------------------------------------------------------------------------
home . archive . source on GitHub
==============================================================================================