==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 15 . TRAINING THE Q-NETWORK . PART 4 OF 5
The World Calls Three Times: Wiring the Agent by Pencil
============================================================================================
This chapter built the pieces that TRAIN a Q-network: a net that reads a spot and
hands back a worth per move; backprop, which walks one miss back to every dial;
Adam, which spends each dial's nudge well; and the replay bin plus the frozen twin,
which grade a whole batch of stored moves against a held-still copy of the net.
Four working parts -- and still no agent. An agent is not a pile of functions; it
is a thing the world can CALL, moment by moment, that answers with a move and
quietly learns on the side. This post wires the parts into that callable thing,
then runs it on the lander.
A worth = the net's guess of total future reward for a move. A move = which engine
to fire (0 left, 1 main, 2 right, 3 coast). A spot (a state) = the lander's eight
dials right now. The replay bin = a list of past moves, each stored as a five-part
note: the spot left, the move taken, the reward, whether that move ENDED the run
(1 yes / 0 no), and the spot landed on.
SO THE WORLD DOES NOT RUN THE AGENT -- IT CALLS IT, AT THREE KINDS OF MOMENT
You do not write a loop that drives the lander. The world (the simulator) runs the
loop, and it reaches into your agent at three kinds of moment: the FIRST moment of
a run, every MIDDLE moment, and the LAST moment. So the agent is three functions
the world calls, not one:
episode timeline (one run of the lander):
first middle middle last
moment moment moment ... moment
| | | |
v v v v
agent_start agent_step agent_step agent_end
(S0) (R1, S1) (R2, S2) (R_last)
At the first moment the world has handed over a starting spot but no reward yet
(nothing has happened). At every middle moment it hands a reward and the new spot
its last move produced. At the last moment the run is over -- it hands the final
reward and there is no new spot. Three moments, three functions.
WHICH MEANS THE FIRST CALL ONLY PICKS A MOVE AND REMEMBERS IT
The first call, agent_start, gets the starting spot S0 and nothing else. There is
no reward to learn from yet, so it does exactly two things: pick a move, and write
down what it did.
agent_start(S0):
pick a move A0 from S0 (softmax over the net's four worths -- below)
remember: last_spot = S0
last_move = A0
hand A0 back to the world
Picking the move: read the net's four worths at S0, turn them into chances with
softmax -- raise e to each worth, divide each by the total, so the four are
positive and add to 1, a bigger worth earning a bigger slice -- then SAMPLE one
engine from those chances (not always the biggest, so no engine goes untried).
Call the sampled engine A0.
The two "remember" lines are the whole subtlety of this post, so look hard at what
is stored:
last_spot = S0 (the spot we were just handed)
last_move = A0 (the move we are about to hand back for the world to execute)
last_move is not a move already carried out. It is the move the world is ABOUT to
carry out next. Why that matters is the next section.
BECAUSE THE NEXT CALL MUST STORE last_move AS THE CAUSE OF THE REWARD IT RECEIVES
When the world calls agent_step, it hands back a reward R1 and a new spot S1. That
reward and that spot are the RESULT of the move the agent last handed over -- A0.
So the note to drop in the bin is "from S0, move A0, got R1, did not end, landed
S1." That is why last_move had to hold A0: it is the cause of the reward now
arriving. Draw the first two middle calls in order:
agent_start: pick A0 remember last_spot=S0, last_move=A0
(world executes A0)
agent_step 1: world gives R1, S1
STORE (S0, A0, R1, 0, S1) <- correct
pick A1 remember last_spot=S1, last_move=A1
(world executes A1)
agent_step 2: world gives R2, S2
STORE (S1, A1, R2, 0, S2) <- correct
Now see the bug that lurks if the "remember" lines are skipped. Suppose agent_step
1 stored its note but forgot to update last_move to A1. Then at agent_step 2,
last_move is still A0, and the stored note would read:
STORE (S1, A0, R2, 0, S2) <- WRONG: R2 came from A1, but A0 is blamed
A reward would be pinned on a move that did not cause it, and the net would learn a
lie. The two "remember" lines exist so every stored note blames the RIGHT move.
That is the entire reason last_move is saved on every call: it must always hold the
move whose reward the next call will receive.
SO A MIDDLE CALL DOES FOUR THINGS: STORE, LEARN, PICK, REMEMBER
With the timing settled, the middle call is straightforward. agent_step gets
(reward R, new spot S), and in order:
agent_step(R, S):
STORE the note (last_spot, last_move, R, ended? 0, S) -> drop in the bin
LEARN run the replay updates (below)
PICK a move A from S (softmax over the net's worths at S, then sample)
REMEMBER last_spot = S , last_move = A
hand A back to the world
The ended? stored is 0, because the run did NOT end on this move -- there is a real
next spot S to read a future from. LEARN is everything the chapter already built,
restated in one breath: photocopy the net into the frozen twin; grab a random batch
of notes from the bin; read each landing's worth through the TWIN and each
left-spot's worth through the live net; subtract to get a how-wrong per note; walk
each how-wrong back to every dial; take one Adam step on the batch's averaged
grades -- and repeat for the several replay passes of this burst. One real move in,
several stored moves rehearsed out.
One gate guards the whole LEARN block: it runs only when the bin holds MORE notes
than one batch draws. The real lander draws batches of 8, so its first eight
note-filing calls learn nothing at all -- the first burst fires on the call that
files note 9, and every call learns from then on. The gate is not caution for its
own sake: drawing 8 notes from a bin of 3 would just re-teach the same few moments
and call them the world. The machine waits until a draw can actually vary.
BUT THE LAST CALL HAS NO NEXT SPOT, SO ITS NOTE IS MARKED ENDED
When the run ends -- the lander touches down or crashes -- the world calls
agent_end with only a final reward R. There is no new spot: the game is over. So
agent_end stores a note whose landing spot is a dead placeholder (a row of zeros),
MARKED with ended? = 1, and returns no move (there is nothing left to do):
agent_end(R):
STORE the note (last_spot, last_move, R, ended? 1, zero_spot) -> drop in the bin
LEARN run the replay updates (same as agent_step)
(return nothing -- the run is over)
That ended? = 1 is the flag the batch machine was built to read. When this note is
later replayed, its target is reward + dilute x (worth of the landing) x
(1 - ended?) = R + dilute x (worth) x 0 = R -- the bare final reward, no faded
future, because there IS no future past the end. A crash's target is its penalty
alone; a landing's target is its bonus alone. The two calls, side by side:
what agent_step (mid-run) agent_end (run over)
---- -------------------- --------------------
world hands reward R, new spot S reward R only
ended? flag 0 (a real next spot follows) 1 (nothing follows)
landing stored S a dead placeholder spot
returns a move for the world to run nothing (episode done)
One whole tiny run, played out. The world makes four calls:
agent_start(S0) -> the agent samples engine 2
agent_step(+5, S1) -> the agent samples engine 1
agent_step(-3, S2) -> the agent samples engine 3
agent_end(-100) (the lander crashed)
Write every note the bin receives, in order -- five parts each. Then: which engine
gets blamed for the -100, and what will that note's target be when it is replayed
(dilute 0.9)?
CHECK: note 1 (at step 1): ( S0 , 2 , +5 , 0 , S1 )
note 2 (at step 2): ( S1 , 1 , -3 , 0 , S2 )
note 3 (at end) : ( S2 , 3 , -100 , 1 , zero-spot )
Three notes from four calls -- agent_start stores nothing (no reward has
happened yet). The -100 blames engine 3 -- the move picked at step 2,
the last move actually executed. Its replayed target: -100 + 0.9 x
(whatever the twin reads off the zero-spot) x (1 - 1) = -100 exactly.
SO STRUNG TOGETHER, THE THREE CALLS ARE A LANDER THAT LEARNS TO FLY
The world runs its own loop and knocks on the three doors; the agent answers and
learns on the side. One whole run reads: agent_start picks the opening move;
agent_step fires for every middle moment -- store, learn, pick, remember -- until
the lander lands or crashes; agent_end stores the final ended note and learns one
last time. Then the world starts the next run and calls agent_start again. Nothing
else drives it.
Run that over a few hundred runs and the learning curve climbs: the early runs
crash (the net's worths are near-random, the softmax nearly a coin-toss), but
every stored move feeds the replay bin, every batch nudges the dials, and the
worths sharpen until the softmax reliably fires the engine that keeps the lander
upright and slows its fall. The pile of parts, wired through these three calls, is
a complete deep-RL agent flying LunarLander -- and the next post assembles every
dial of it, end to end, on one page.
WHERE IS MY LOOP? -- THE WIRING MISTAKES BEFORE THE FIRST CLEAN EPISODE
"Where is MY loop? I kept writing a main loop to drive the lander." But the
simulator already owns the loop -- it calls agent_start once, agent_step a few
hundred times, agent_end once, and there is no place for a second driver. My job
is three door-answers, not an engine room. The world calls; the agent only ever
ANSWERS. If you are writing a while-loop around the lander, you are building a
second world.
"agent_start should store a note too -- it did something!" Count what the first
call holds: a note has five parts, and at that moment I hold exactly one of them
(S0). No reward has been paid, nothing has ended, nothing has been landed on. The
drill above: four calls, three notes. A note records a CONSEQUENCE, and the first
call has none yet; its move gets its note at the NEXT call.
"The zero-spot placeholder will poison the learning -- the twin reads worths off
a fake spot!" It does read them -- say it reads [4, 4, 4, 4] off the zeros -- and
then the target multiplies that whole future by (1 - ended?) = 0, so the fat 4s
vanish and the target is the bare reward. The placeholder exists only so the note
keeps five parts; its worths are born condemned. Do not fear a garbage value that
a mask is about to erase; fear a missing flag.
"I stored the note AFTER picking and remembering." Then the note read
(S, A_new, R, ...) -- the fresh pick blamed for the old reward, the same lie as
the forgotten remember-lines, one line later. The order inside a middle call is
fixed and means something: STORE first (the old pair earns its note while it
still IS the old pair), LEARN, then PICK, then REMEMBER the new pair. Two of the
four would happily swap; those two never can.
SEAM. Pencil ends here; below, the same numbers in Python.
dilute = 0.9
batch_size = 8
# four world calls; trace last_spot and last_move at each
last_spot, last_move = "S0", 2 # agent_start: starting spot, pick engine 2
# agent_step 1: world gives R=+5, lands S1
note1 = (last_spot, last_move, 5, 0, "S1") # (S0, eng2, +5, 0, S1)
last_spot, last_move = "S1", 1 # pick engine 1, remember
# agent_step 2: world gives R=-3, lands S2
note2 = (last_spot, last_move, -3, 0, "S2") # (S1, eng1, -3, 0, S2)
last_spot, last_move = "S2", 3 # pick engine 3, remember
# agent_end: world gives R=-100, no new spot
note3 = (last_spot, last_move, -100, 1, "zeros") # (S2, eng3, -100, 1, zeros)
print(note1) # ('S0', 2, 5, 0, 'S1')
print(note2) # ('S1', 1, -3, 0, 'S2')
print(note3) # ('S2', 3, -100, 1, 'zeros')
# note 3 target when replayed: ended?=1 kills the future piece no matter what twin reads
twin_reads = 4.0 # placeholder off zero-spot
target3 = -100 + dilute * twin_reads * (1 - 1) # -100+0.9*4*0 = -100
print(target3) # -100.0
# bin gate: first learn fires when bin holds more than one batch
bin_len = len([note1, note2, note3]) # 3 in this tiny run
print(bin_len > batch_size) # False -- too few; no learning yet
print(9 > batch_size) # True -- first learn at note 9 in real lander
# wrong-order bug: forgot to update last_move after step 1
wrong_note2 = ("S1", 2, -3, 0, "S2") # engine 2 blamed instead of 1
print(wrong_note2 == note2) # False -- wrong blame
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 15 -- Training the Q-Network):
Part 1 -- Backpropagation by Pencil .
Part 2 -- Adam and Replay by Pencil .
Part 3 -- The Frozen Twin: A Batch of Misses at Once .
Part 4 (this post) .
Part 5 -- From Eight Dials to a Soft Landing: The Whole Agent by Pencil -- the chapter closes here
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================