==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 15 . TRAINING THE Q-NETWORK . PART 3 OF 5
The Frozen Twin: A Batch of Misses at Once
============================================================================================
The previous post grabbed a random handful of past moves out of a bin -- a BATCH --
and said: for each one, rebuild its target with the current net, then take one Adam
step on the averaged miss. It waved at two things to keep moving. First, it never
showed the batch worked as a batch -- all the misses computed in one sweep of
arithmetic, not one move at a time. Second, "rebuild with the current net" is not
quite true. The net that GRADES the batch is a frozen copy, held still on purpose,
and holding it still is the whole point of this post.
A quick redraw so nothing leans on memory. The bin is a list of past moves; each is
a five-part note: the spot you left s, the move you took a, the reward r, whether
that move ENDED the game (1 yes / 0 no -- a crash or a landing), and the spot you
landed on s'. To learn, you scoop a batch out at random. Take a batch of two:
the batch (two notes pulled from the bin):
# s (left) a (took) r ended? s' (landed)
-- -------- -------- -- ------ -----------
1 s1 1 = main +2 0 s1' (still flying)
2 s2 2 = right -1 1 s2' (crashed -- game over)
There are four engines to choose between -- 0 = left, 1 = main, 2 = right, 3 =
coast -- so a worth-row has four numbers. Note 1 fired the MAIN engine and lived;
note 2 fired the RIGHT engine and crashed. The job: one how-wrong number per note.
The rest is where each number comes from.
TARGET NEEDS THE LANDING'S WORTH, BUT WHICH MOVE COMES NEXT?
A target is reward-now plus the faded worth of where you landed:
target = r + dilute x (worth of the landing s') dilute = 0.9
But "worth of the landing" is worth-of-WHAT-move? You have not chosen the next move
yet. So do not guess one -- average over ALL four next-moves, each weighted by how
likely the habit is to pick it. That average-over-the-odds is the Expected-Sarsa
worth of a spot. Draw the frozen twin (call it current_q for now -- the "why
frozen" is the last section) reading both landing spots at once, one row out per
note:
s1' -> +-----------+ -> [ 0 , 2 , 0 , 0 ] worth of each engine at landing 1
| current_q |
s2' -> +-----------+ -> [ 4 , 4 , 4 , 4 ] worth of each engine at landing 2
q_next = [ 0 2 0 0 ] (one row per note, one column per engine: a 2 x 4 block)
[ 4 4 4 4 ]
Now the odds. The habit here turns a row of worths into a row of chances with
softmax: raise e (the constant 2.71828...) to each worth, then divide each by the
row's total, so the four chances are all positive and add to 1, and a bigger worth
earns a bigger slice. Row 1, worths [0, 2, 0, 0]:
e^0 = 1 e^2 = 7.39 e^0 = 1 e^0 = 1 total = 1 + 7.39 + 1 + 1 = 10.39
chances = [ 1/10.39 , 7.39/10.39 , 1/10.39 , 1/10.39 ] ~ [ 0.096 , 0.711 , 0.096 , 0.096 ]
(Check: 0.096 + 0.711 + 0.096 + 0.096 = 0.999, a rounded 1.) Row 2, worths
[4, 4, 4, 4]: every e^4 is the same, so the four slices are equal --
[0.25, 0.25, 0.25, 0.25].
chances = [ 0.096 0.711 0.096 0.096 ] (softmax of each worth-row)
[ 0.25 0.25 0.25 0.25 ]
The spot's worth is then chances-dotted-into-worths, row by row -- each engine's
worth times its own chance, summed:
row 1: 0.096x0 + 0.711x2 + 0.096x0 + 0.096x0 = 1.422 ~ 1.42
row 2: 0.25x4 + 0.25x4 + 0.25x4 + 0.25x4 = 4.0
v_next = [ 1.42 , 4.0 ] (one worth per landing spot)
One more landing row. The twin reads worths [1, 1, 3, 1] at some
landing. Softmax it (e^1 = 2.72, e^3 = 20.09) and dot the chances back into the
worths.
CHECK: total = 2.72 + 2.72 + 20.09 + 2.72 = 28.25
chances ~ [ 0.096 , 0.096 , 0.711 , 0.096 ] -- the SAME chances as the
[0,2,0,0] row above! [1,1,3,1] is that row plus 1 everywhere, and softmax
only reads the GAPS between worths, not their heights.
worth = 0.096x1 + 0.096x1 + 0.711x3 + 0.096x1 = 0.288 + 2.133 ~ 2.42
-- one higher than that row's 1.42, because every worth is one higher.
BUT A CRASH HAS NO LANDING WORTH, SO ZERO THE ENDED ONES
That 4.0 is a trap. Note 2 CRASHED -- the game ended -- so there is no future to
fade in. Its s' is a dead placeholder, and the frozen twin does not know that; it
happily reads a fat 4.0 off it. Kill those by hand with the ended? flag: multiply
each worth by (1 - ended?), which is 1 for a move still in play and 0 for a move
that ended:
ended? = [ 0 , 1 ]
1 - ended? = [ 1 , 0 ]
v_next x (1 - ended?) = [ 1.42 x 1 , 4.0 x 0 ] = [ 1.42 , 0 ]
See why it matters. The target is reward-now plus dilute times that landing worth:
target = r + dilute x [ landing worth after the zeroing ]
note 1: 2 + 0.9 x 1.42 = 2 + 1.278 = 3.278 ~ 3.28
note 2: -1 + 0.9 x 0 = -1 + 0 = -1.0
target = [ 3.28 , -1.0 ]
Had you skipped the (1 - ended?), note 2's target would be -1 + 0.9 x 4.0 = +2.6 --
a crash scored as a REWARD, because the dead placeholder's worth leaked into it.
The zeroing is what keeps a game-ending move graded on its final reward alone.
PREDICTION IS THE MOVE YOU REALLY MADE, SO PICK JUST THAT ONE
The target is what the move SHOULD have been worth. Now the guess: what did the net
actually think of the engine you fired? This time the LIVE net -- the one being
trained, call it network -- reads the two spots you LEFT, again one row out per
note:
s1 -> +---------+ -> [ 1.0 , 2.5 , 0.5 , 1.5 ] network's worths at spot 1
| network |
s2 -> +---------+ -> [ 0.0 , 1.0 , 0.5 , -0.5 ] network's worths at spot 2
q_all = [ 1.0 2.5 0.5 1.5 ] (2 x 4: a worth for every engine at every left-spot)
[ 0.0 1.0 0.5 -0.5 ]
Here is the asymmetry worth pausing on. On the target side you did NOT know the
next move, so you averaged all four columns. Here you DO know the move -- it is
stored in the note -- so you pick exactly ONE column and drop the other three.
Note 1 fired engine 1 (main); note 2 fired engine 2 (right). Read down each note's
own taken-column:
note 1, took engine 1: row [1.0, 2.5, 0.5, 1.5] -> pick column 1 -> 2.5
note 2, took engine 2: row [0.0, 1.0, 0.5,-0.5] -> pick column 2 -> 0.5
q_took = [ 2.5 , 0.5 ] (the worth of the engine actually fired, one per note)
SUBTRACT, AND THE WHOLE BATCH'S MISS FALLS OUT AT ONCE
Target minus guess, straight across -- how wrong the live net was on each note you
replayed:
how-wrong = target - q_took
note 1: 3.28 - 2.5 = 0.78 (net lowballed the main engine -- pull it up)
note 2: -1.0 - 0.5 = -1.5 (net overrated the right engine that crashed -- push down)
how-wrong = [ 0.78 , -1.5 ]
That two-number row is the batch's miss, computed in one sweep rather than one note
at a time. Each number then rides the backprop post's machine: walk it backward
through the fired engine's slot to every dial, summing pulls; the two notes' grades
are averaged into the single Adam step of the previous post. A batch of misses, one
step. Two pieces are still hand-waved: how one dial obeys two notes at once, and
the word "frozen."
A third note joins the batch: (s3, took engine 0, reward 0.5,
ended? 0, s3'). The frozen twin reads the landing s3' as [2, 2, 2, 2]; the live
net reads the left-spot s3 as [1.8, 0, 0, 0]. dilute 0.9. Compute the landing's
worth, the target, the guess, and the miss.
CHECK: equal worths -> equal chances [0.25 x 4] -> landing worth = 2.0
still flying -> x (1 - 0) keeps the 2.0
target = 0.5 + 0.9 x 2.0 = 2.3
guess = pick column 0 of [1.8, 0, 0, 0] = 1.8
miss = 2.3 - 1.8 = 0.5 (lowballed -- pull engine 0 up a little)
ONE DIAL, TWO NOTES PULLING OPPOSITE WAYS -- SO WHY DO THEIR VOTES ADD?
Now pay the first hand-wave. Every dial in the net served BOTH notes' forward
passes -- the same sheets read s1 and s2. So when the misses ride backward, one
dial receives TWO verdicts, and nothing says they agree. Take one dial d,
currently 1.0, and say the backward walks (run once per note) return:
note 1: d's pull on note 1's guess = 2 note 1's miss = +0.78
note 2: d's pull on note 2's guess = 3 note 2's miss = -1.5
Note 1 was lowballed and wants its guess UP: its vote on d is
(+0.78)(2) = +1.56. Note 2 overrated an engine that crashed and wants its
guess DOWN: its vote is (-1.5)(3) = -4.5. One dial cannot move two directions
at once. What it does instead is forced by a one-line theorem, derived on the
spot. The batch's wrongness is the SUM of the notes' wrongnesses (averaging is
the same sum, divided by the note count). Wiggle d by a hair h: note 1's
wrongness moves by (its slope) x h, note 2's moves by (its slope) x h, so the
sum moves by (slope1 + slope2) x h. The slope of a sum is the sum of the
slopes -- nothing deeper. So votes on a shared dial ADD:
combined vote = (+1.56) + (-4.5) = -2.94 averaged over 2 notes: -1.47
nudge = size x (-1.47) (size 0.01: d moves by -0.0147)
The louder note wins -- not by silencing the other, but by outvoting it. The
dial falls: note 2's fat error shrinks a lot, note 1's small hope is set back
a little, and the batch's TOTAL wrongness falls by the most this one dial can
manage. The rule in five words: multiply along, add across. Multiply pulls
along one note's chain; add the votes across the notes that share the dial.
The neatest special case is a nudge. A nudge's pull on its own note's raw
score is exactly 1 -- raise the nudge a hair and the score rises a hair,
nothing dilutes it. So a shared nudge's combined vote is each note's arriving
blame times 1, added: here, if both notes' blame reached the same nudge,
(+0.78) + (-1.5) = -0.72. One number was handed out to every note on the way
forward; every note's verdict is added up on the way back. Handed out forward,
summed backward -- the pairing is exact, and it is why a batch of 32 notes
hands each of its shared nudges a 32-term sum.
WHY TWO NETS -- THE ONE THAT GRADES MUST HOLD STILL
Look back at what read what. The LANDINGS (for the target) went through current_q,
the frozen twin. The LEFT-SPOTS (for the guess) went through network, the live one.
Two copies of the same net, one held still. Here is the reason, and it hides inside
a loop the previous post mentioned once: after a single real move, you do not learn
just once -- you scoop a batch and take several replay passes over fresh batches
before stepping back into the world (say four). Each pass changes the live net's
dials. Draw the passes:
before the passes: network = V1 ---- photocopy ----> current_q = V1
pass 1: network V1 -> V2 current_q still V1
pass 2: network V2 -> V3 current_q still V1
pass 3: network V3 -> V4 current_q still V1
pass 4: network V4 -> V5 current_q still V1
Now suppose you had ONE net doing both jobs. Pass 1 grades against V1 and nudges
toward it. Pass 2 would grade against V2 -- a target that just moved because you
moved it. Pass 3 grades against V3. The thing you are chasing slides every pass,
exactly as fast as you chase it:
one net: guess V1 -> aim at target(V1), then net becomes V2
guess V2 -> aim at target(V2), then net becomes V3 <- the target keeps fleeing
guess V3 -> aim at target(V3), ...
You cannot hit a mark that jumps each time you shoot. So the code photocopies the
net BEFORE the passes begin and freezes the copy as current_q. Every pass reads its
target off that same V1 while only the live net moves. The mark holds still; the
shots land. When the four passes end and the next real move comes, a new photocopy
is taken from wherever the live net now sits, and the next burst of passes aims at
THAT. The twin is refreshed between bursts, frozen within one.
So the one line the previous post rounded off -- "rebuild the target with the
current net" -- reads exactly now: rebuild the target with the FROZEN TWIN, the
copy taken before this burst of passes. The guess uses the live net; the target
uses the twin; and because the twin does not budge while the batch is swept, the
miss you compute is a miss against a fixed mark, not a mirage.
ONE NET FOR BOTH JOBS, AND A MARK THAT KEPT FLEEING
"One net can do both jobs -- why keep a copy?" My misses refused to shrink across
the replay passes, and I blamed the step size first. The real culprit: with one
net, every pass moves the mark by the very step just taken -- guess V1 aims at
target(V1) and becomes V2; V2 aims at target(V2); the gap you are closing reopens
as fast as you close it. Freeze the twin and all four passes aim at the same V1
mark. The grader holds still inside a burst; it is refreshed only between bursts.
"Softmax blew up on big worths." A worth-row like [500, 502, 500, 500] asks for
e^502 -- a number with 219 digits; the machine chokes. But softmax reads only the
GAPS. Subtract the row's biggest worth first: [500, 502, 500, 500] - 502 =
[-2, 0, -2, -2], and e^-2 = 0.135, e^0 = 1, total 1.41, chances
[0.096, 0.711, 0.096, 0.096] -- the exact same chances as the row [0, 2, 0, 0].
Nothing changed but the arithmetic stopped exploding. Always shift a row so its
biggest worth is 0 before raising e to it. The chances cannot tell the
difference; the machine can.
"I gathered the TAKEN column on the target side too." Symmetry felt right: pick
the stored move everywhere. On note 1 that reads q_next's column 1 -> worth 2,
instead of the averaged 1.42 -- but the stored move is the move you took at s,
and nothing says the habit will repeat it at s'. The future is unchosen, so it
gets the AVERAGE; the past is on record, so it gets the PICK. Average where you
do not know (the landing), pick where you do (the move taken). The asymmetry is
the design.
"I wrote 'the twin is the net' instead of photocopying it." One line of
bookkeeping -- point a second name at the same sheets -- and my "frozen" twin
moved with every nudge, because both names were holding the same paper: after
pass 1 the "twin" already read V2, and the fleeing-mark disease came back wearing
a twin costume. The twin is a PHOTOCOPY -- a full copy of every dial made before
the burst -- never a second label on the live sheets. (In the code below, that is
deepcopy, and it is the whole reason the word appears.)
SEAM. Pencil ends here; below, the same numbers in Python.
import math
dilute = 0.9
# frozen twin reads landings
# note 1 landing s1': worths [0,2,0,0]
e0, e2 = math.exp(0), math.exp(2) # 1.0 7.389
tot1 = e0 + e2 + e0 + e0 # 10.389
c10, c11 = e0/tot1, e2/tot1 # 0.096 0.711
v1 = c10*0 + c11*2 + c10*0 + c10*0 # 0.711*2 = 1.422
print(round(c10,3), round(c11,3), round(v1,3)) # 0.096 0.711 1.422
# note 2 landing s2': worths [4,4,4,4] -> equal chances
v2 = 0.25*4 + 0.25*4 + 0.25*4 + 0.25*4 # 4.0
print(v2) # 4.0
# zero the crashed note (ended?=[0,1])
v1_safe = v1 * (1 - 0) # 1.422*1 = 1.422
v2_safe = v2 * (1 - 1) # 4.0*0 = 0.0
print(round(v1_safe,3), v2_safe) # 1.422 0.0
# targets
t1 = 2 + dilute * v1_safe # 2+0.9*1.422 = 3.28
t2 = -1 + dilute * v2_safe # -1+0.9*0 = -1.0
print(round(t1,3), t2) # 3.28 -1.0
# live network: pick taken-engine column
# note 1 took engine 1 -> col 1 of [1.0,2.5,0.5,1.5] = 2.5
# note 2 took engine 2 -> col 2 of [0.0,1.0,0.5,-0.5] = 0.5
how_wrong1 = t1 - 2.5 # 3.28-2.5 = 0.78
how_wrong2 = t2 - 0.5 # -1.0-0.5 = -1.5
print(round(how_wrong1,3), how_wrong2) # 0.78 -1.5
# YOUR TURN: note 3 (engine 0, reward=0.5, ended?=0, landing [2,2,2,2])
v3 = 0.25*2 + 0.25*2 + 0.25*2 + 0.25*2 # 2.0
t3 = 0.5 + dilute * v3 # 0.5+0.9*2 = 2.3
miss3 = t3 - 1.8 # 2.3-1.8 = 0.5
print(v3, t3, round(miss3,4)) # 2.0 2.3 0.5
# softmax stability: [1,1,3,1] gives same chances as [0,2,0,0]
e1, e3 = math.exp(1), math.exp(3) # 2.718 20.086
tot3 = e1 + e1 + e3 + e1 # 28.24
cs, cf = e1/tot3, e3/tot3 # 0.096 0.711
worth3 = cs*1 + cs*1 + cf*3 + cs*1 # 0.288+2.133 = 2.42
print(round(cs,3), round(cf,3), round(worth3,2)) # 0.096 0.711 2.42
# ONE SHARED DIAL, TWO VOTES: multiply along a note's chain, ADD across notes
vote1 = 0.78 * 2 # note 1: miss +0.78 x pull 2 = +1.56 (wants d up)
vote2 = -1.5 * 3 # note 2: miss -1.5 x pull 3 = -4.5 (wants d down)
combined = vote1 + vote2 # 1.56 - 4.5 = -2.94 the slope of a sum is the sum of slopes
avg = combined / 2 # -1.47 averaged over the 2 notes
print(vote1, vote2, combined, avg) # 1.56 -4.5 -2.94 -1.47
print(0.01 * avg) # -0.0147 the ONE nudge d actually takes -- the louder note outvoted
print(0.78 + (-1.5)) # -0.72 a shared NUDGE's combined vote: each miss x pull 1, summed
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 15 -- Training the Q-Network):
Part 1 -- Backpropagation by Pencil .
Part 2 -- Adam and Replay by Pencil .
Part 3 (this post) .
Part 4 -- The World Calls Three Times: Wiring the Agent by Pencil .
Part 5 -- From Eight Dials to a Soft Landing: The Whole Agent by Pencil
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================