==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 15 . TRAINING THE Q-NETWORK . PART 1 OF 5
One Miss, a Thousand Nudges: Backpropagation by Pencil
============================================================================================
The previous chapter closed on a sting. A network guessed an engine was worth 5.5;
the grading target said 4.42; the miss was one number, -1.08. And that one number
has to fix a whole pile of dials.
On a tile-coded car the miss had a pointer: the feature lit exactly the tiles to
nudge, and you were done. A network has no such pointer. The miss is born at the
output, but the dials that caused it sit two sheets back, behind a bend. This post
walks that one number backward to every dial, by pencil -- which is the whole of
what backpropagation is.
MACHINE AND MISS, REBUILT FROM SCRATCH
Nothing here leans on memory. The toy network turns a 2-number state into 4
engine-worths (engines numbered 0 to 3), through one middle row of 3 numbers, with
a bend in between:
state s sheet A middle psi bend x sheet C worths Q
[s1 s2] --> (2 x 3) --> [p1 p2 p3] --> max(.,0) --> --> (3 x 4) --> [Q0..Q3]
1 x 2 + b1 1 x 3 1x3 + b2 1 x 4
With these dials and the state s = [1, 2]:
A = [ 1 -2 0.5 ] b1 = [ 0.5 -0.5 1.0 ] C = [ 2 -1 0 1 ] b2 = [0.5 1.0 0.0 -0.5]
[ 0.5 1 -1 ] [ 1 1 -1 0 ]
[ -1 2 1 -1 ]
the forward pass runs (each middle number = state dotted with a column of A, then
the bend flattens negatives, then each worth = the bent row dotted with a column
of C):
psi = s A + b1 = [ 2.5, -0.5, -0.5 ]
x = max(psi, 0) = [ 2.5, 0, 0 ] <- two middle numbers are DEAD (were negative)
Q = x C + b2 = [ 5.5, -1.5, 0.0, 2.0 ]
We fired engine 0 (worth Q0 = 5.5). The grading move -- reward plus the diluted,
chance-weighted read of the next spot, built in the previous post -- returned a
target of 4.42 for that engine, so the miss, the same "how wrong" as every post, is
how wrong = target - worth fired = 4.42 - 5.5 = -1.08
written as a row that is zero except in the fired engine's slot: [ -1.08, 0, 0, 0 ].
The guess was 1.08 too HIGH, so every dial that pushed Q0 up wants pulling down a
little.
ONE NUMBER CANNOT INSTRUCT A SHEET -- SO ASK EACH DIAL WHAT IT DID
Sheet C alone is 12 dials; sheet A is 6; with the two bias rows that is 25 dials on
this toy (3,332 on the real lander: 8 dials -> 256 middles -> 4 engines). You
cannot hand "-1.08" to 25 dials and call it an instruction -- they did not all push
equally. Watch the lazy answer fail on just the four dials that feed Q0 directly
(sheet C's first column, values 2, 1, -1, plus its bias 0.5, fed by the bent row
x = [2.5, 0, 0]):
hand the whole -1.08 to each of the four: Q0 moves by (2.5 + 0 + 0 + 1) x (-1.08) = -3.78
-- asked to drop 1.08, dropped 3.5 times that
and two of the four moved for NOTHING: their middles are dead (the second and third
numbers of x are 0), so changing those two dials cannot touch Q0 today -- yet WILL
corrupt tomorrow's states, where those middles wake up.
Worse: even among splits that land the drop exactly, there are infinitely many. Put
it all on the bias (lower it 1.08); put it all on the top dial (lower it
1.08 / 2.5 = 0.432); any mixture in between. Every one of them repairs THIS state's
guess. The machine needs a reason to pick ONE.
The reason is the fair-share rule that has run this whole blog, first met where a
one-hot feature steered a nudge onto one weight (nudge each weight by how-wrong
times its feature). Here the same rule, one word swapped: nudge each dial in
proportion to HOW MUCH IT MOVED THE THING WE GRADED. The repair then lands where
the cause was -- big pushers pay big shares, dials that could not touch Q0 are left
alone -- so everything the dials know about OTHER states is disturbed as little as
possible while this miss shrinks.
So for every dial ask one question: if I raise this dial a hair, how much does Q0
(the fired engine's worth) rise? Call that number the dial's PULL on Q0. Then nudge
the dial by
dial = dial + size x (how wrong) x (its pull on Q0)
A big puller takes a big share of the -1.08; a dial with zero pull is left alone.
The pull has to be CHASED back through two sheets and a bend -- that chase is the
rest of this post.
SO WALK THE MISS BACKWARD, OUTPUT FIRST
Start where the miss is born: the worths Q. Only the fired engine matters -- Q0 --
so its pull on ITSELF is 1, and the other three worths are bystanders, pull 0:
pull on the worths: [ 1, 0, 0, 0 ] (Q0 cares about Q0; Q1,Q2,Q3 are off the hook)
This single 1-in-a-row is the seed. Every pull below is this seed multiplied
backward.
WHICH HANDS SHEET C ITS SHARE -- AND ONLY THE FIRED COLUMN MOVES
A worth is built as Q_k = (x dotted with column k of C) + b2_k. So the dial C[i,k]
is multiplied by x_i on its way into worth k. Raise C[i,k] by a hair and worth k
rises by x_i. Its pull is therefore x_i -- but ONLY on worth k, and we only care
about worth 0 (the fired engine's column). Every other column of C feeds a
bystander worth, pull 0:
pull on C[i, fired col] = x_i pull on every other column of C = 0
With x = [2.5, 0, 0], the pull-sheet for C is x dropped into the fired column,
zeros else:
pull on C : col0 col1 col2 col3 (col0 = fired engine 0)
row0 [ 2.5 0 0 0 ] row i = hidden number x_i
row1 [ 0 0 0 0 ]
row2 [ 0 0 0 0 ]
Two things to see. First, the "one slot" from the miss has become "one column" --
only the fired engine's column of C will ever move; the other three sleep. Second,
rows 1 and 2 are zero because x2 = x3 = 0: the dead middle numbers carried nothing
in, so their dials get no blame. The bias b2 adds straight onto the worths, so its
pull is the seed itself:
pull on b2 = [ 1, 0, 0, 0 ]
WHICH MEANS THE BEND DECIDES WHAT GETS THROUGH
To reach sheet A we must pass back through the middle numbers x, and through the
bend that made them. First, how much does each middle number x_i pull on Q0? In
Q0 = x dotted with column 0 of C, the number x_i is multiplied by C[i,0], so its
pull is C[i,0] -- column 0 of C, which is [2, 1, -1]:
pull on x = column 0 of C = [ 2, 1, -1 ]
But x came from psi through the bend x = max(psi, 0). A middle number that was
POSITIVE passed straight through (a hair up in psi is a hair up in x, pull 1); one
that was flattened to zero is stuck flat (a hair up in psi, still zero out, pull
0). So the bend is a gate, open where psi was positive, shut where it was not.
With psi = [2.5, -0.5, -0.5]:
bend gate = [ 1, 0, 0 ] (open at p1; shut at the two that went negative)
Multiply the pull-on-x by the gate to get the pull on the pre-bend numbers psi:
pull on psi = pull on x x gate
= [ 2, 1, -1 ] x [ 1, 0, 0 ]
= [ 2, 0, 0 ]
The dead middle numbers slam the gate: whatever blame was heading for them (the 1
and the -1) hits the floor. Only the live number p1 carries blame onward.
WHICH HANDS SHEET A ITS SHARE -- THROUGH THE ONE LIVE NUMBER
Last sheet. A middle number is built psi_j = (s dotted with column j of A) + b1_j,
so the dial A[i,j] is multiplied by s_i on its way into psi_j. Raise A[i,j] a hair
and psi_j rises by s_i; but psi_j only matters as much as psi_j pulls on Q0, which
we just found. So the pull is "input s_i times the blame on psi_j" -- every
(input, blame) pair multiplied:
pull on A[i,j] = s_i x (pull on psi_j)
With s = [1, 2] and pull on psi = [2, 0, 0]:
pull on A : col0 col1 col2
row0 [ 1 x 2 = 2 0 0 ] (row i = input dial s_i)
row1 [ 2 x 2 = 4 0 0 ]
Only column 0 -- the live middle number's column -- carries anything; the dead two
leave four of sheet A's six dials untouched. And the bias b1 adds straight onto
psi, so its pull is the blame on psi itself:
pull on b1 = [ 2, 0, 0 ]
Notice the shape that keeps repeating: the pull-sheet for a stage is (the input
that fed that stage) times (the blame arriving at that stage), one product per dial
-- "input times signal." For sheet C it was x times the seed; for sheet A it was s
times the blame on psi. That single shape, applied stage by stage from the output
back, IS backpropagation.
Run the whole backward walk again, same state, same dials, but this
time the lander fired ENGINE 3 (worth Q3 = 2.0 on the forward pass above). The
seed becomes [0, 0, 0, 1]. Chase it: pull on C, pull on b2, pull on x (column 3 of
C is [1, 0, -1]), the gate (psi = [2.5, -0.5, -0.5] as before), pull on psi, pull
on A, pull on b1.
CHECK: pull on C = x dropped into column 3: C[0,3] gets 2.5, all else 0
pull on b2 = [ 0, 0, 0, 1 ]
pull on x = column 3 of C = [ 1, 0, -1 ]
gate = [ 1, 0, 0 ] -> pull on psi = [ 1, 0, 0 ]
pull on A = s times that: row0 [1x1=1, 0, 0], row1 [2x1=2, 0, 0]
pull on b1 = [ 1, 0, 0 ]
Same doors, different column: the fired column of C changes, the gate
does not care which engine fired.
BUT IS A PULL REALLY THE RIGHT NUMBER? WIGGLE ONE DIAL AND SEE
Those pulls were chased back through two sheets and a bend by algebra, and algebra
can hide a slipped sign. Before spending the miss on them, check one the blunt way
-- move a single dial a hair and watch the fired worth by brute force. A pull was
DEFINED as "if I raise this dial a hair, how much does Q0 rise," so that is exactly
what to measure.
Take A[1,0] -- row 1, column 0 of sheet A, currently 0.5 -- the dial the backward
walk said pulls hardest, with a pull of 4. Nudge it a whisker, 0.5 -> 0.501, leave
every other dial put, and run the forward pass again. Only the first middle number
uses column 0 of A, so only it changes:
psi1 = (1)(1) + (2)(0.501) + 0.5 = 1 + 1.002 + 0.5 = 2.502 (still positive: gate open)
x1 = 2.502
Q0 = (2.502)(2) + 0.5 = 5.004 + 0.5 = 5.504
Q0 climbed from 5.5 to 5.504 -- a rise of 0.004 for a wiggle of 0.001. Rise per
unit wiggle is 0.004 / 0.001 = 4, the very pull the algebra claimed. The backward
walk was no trick: a pull IS how far the worth moves when you lean on the dial.
Check a second pull the same blunt way. The walk said C[0,0] (currently
2.0) has pull 2.5. Wiggle it to 2.001, rerun only what changes (x = [2.5, 0, 0]
stays put -- sheet A was not touched), and compute rise over wiggle.
CHECK: Q0 = (2.5)(2.001) + 0.5 = 5.0025 + 0.5 = 5.5025
rise = 5.5025 - 5.5 = 0.0025 ; 0.0025 / 0.001 = 2.5. It checks.
With two pulls confirmed by hand, trust the rest and spend the miss.
SO NUDGE EVERY DIAL BY ITS SHARE OF THE MISS
Now spend the miss. Pick a small step size 0.01 (small because many dials move at
once, and a big middle pull like 4 would overshoot). The nudge for each dial is
dial = dial + size x (how wrong) x (its pull) = dial + 0.01 x (-1.08) x (pull)
= dial + (-0.0108) x (pull)
Only dials with non-zero pull move. Worked, every one that does:
A[0,0] : 1.0 + (-0.0108)(2) = 1.0 - 0.0216 = 0.9784
A[1,0] : 0.5 + (-0.0108)(4) = 0.5 - 0.0432 = 0.4568
b1[0] : 0.5 + (-0.0108)(2) = 0.5 - 0.0216 = 0.4784
C[0,0] : 2.0 + (-0.0108)(2.5) = 2.0 - 0.027 = 1.973
b2[0] : 0.5 + (-0.0108)(1) = 0.5 - 0.0108 = 0.4892
Every other dial in A, C, b1, b2 had pull 0 and stands still. The biggest move went
to A[1,0] -- the dial carrying the largest input (s2 = 2) into the only live middle
number -- exactly the fair share: the hardest pusher takes the biggest correction.
Did it work? Run the forward pass again with the nudged dials:
psi = [ 1x0.9784 + 2x0.4568 + 0.4784 , -0.5 , -0.5 ] = [ 2.3704, -0.5, -0.5 ]
x = max(psi, 0) = [ 2.3704, 0, 0 ]
Q0 = 2.3704 x 1.973 + 0.4892 = 4.6768 + 0.4892 = 5.166 (to three decimals)
The fired engine's worth fell from 5.5 toward the target 4.42 -- to 5.166, a step
of about 0.33, not the whole way. One miss, one gentle nudge to two-dozen dials,
the guess a little less wrong. Do this on every step of every episode and the
sheets slowly bend into worths that need almost no correction. That is a network
learning.
WHAT LEARNED AND WHAT SLEPT
Read back what just moved, because the zeros are the lesson. Of 25 dials, FIVE
changed: the fired engine's column of C (one live entry), that column's bias, the
live middle number's column of A (two entries), and its bias. Everything else
slept -- the other three engine columns (we did not fire them) and the two dead
middle numbers (the bend shut their gate). The miss flowed back along exactly the
path it came forward, and nowhere else. Credit, and blame, went only where they
were earned.
DERIVATIVE MEANS RISE PER UNIT WIGGLE -- AND FIVE MORE BLOCKS THAT CLEARED
"Partial derivatives... so this needs real calculus, which I don't have." The WORD
stopped me a full evening before the thing did. Then one division dissolved it:
wiggle A[1,0] by 0.001 and Q0 rises by 0.004; the ratio 0.004 / 0.001 = 4 IS the
partial derivative -- rise per unit wiggle, holding the other dials still. Nothing
more was ever meant by the words. Whenever a symbol frightens, ask what single
division it stands for.
"The gate is 1 or 0 -- so which is it AT the fold, when a middle number lands
exactly on zero?" I waved this off as pedantry until I aimed the wiggle ruler at
it and got two answers. Put psi = 0 exactly, so the bend outputs max(0, 0) = 0.
Wiggle UP by a hair: max(0, 0.001) = 0.001 -- the output moved a full hair,
rise per unit wiggle = 0.001 / 0.001 = 1. Wiggle DOWN by a hair:
max(0, -0.001) = 0 -- the output moved nothing, rise per unit wiggle = 0. The
two probes disagree: 1 from the right, 0 from the left. So at the fold there
IS no single honest rise-per-unit-wiggle -- the bend's graph has a sharp corner
there, and a corner owns no one slope. The machine still needs SOME number to
multiply by, so the gate rule quietly legislates one: "open only if psi is
strictly positive" scores the fold as 0 (the code's p1 > 0 below does exactly
this). Scoring the fold as 1 instead trains just as well. The choice never
matters in practice, because a middle number computed from real-world decimals
lands EXACTLY on 0.000000 about never -- this post's own psi row came out
[2.5, -0.5, -0.5], nowhere near the fold. Rule to carry: the fold is the one
point where the slope question has no answer; pick a side, write it down, and
move on -- but KNOW it has no answer, and know your side was a choice. "Smooth
everywhere except one corner" is the bend's exact shape, and saying it
precisely costs nothing.
"How can I compute pulls FIRST, when I don't yet know what was wrong?" The order
felt backwards -- surely blame needs a crime. But A[1,0]'s pull is 4 no matter what
the target says: the pull is a fact about the MACHINE (wiggle in, rise out), not
about the miss. Only the SPENDING needs the miss: nudge = 0.01 x (-1.08) x 4.
Pulls first (the machine's anatomy), miss second (the day's verdict), product last
(the correction).
"The bias slides Q0 one-for-one -- rise per unit wiggle exactly 1 -- so I wrote
down 1 and called that dial finished." Half a chain, wearing the full chain's
name. The 1 answers one question only: how much Q0 moves per wiggle of b20 -- a
fact about the machine's wiring. The number the correction spends answers a
different question: how much the day's wrongness moves per wiggle of b20 -- and
Q0's own move still has to be priced by the miss riding in from the output:
1 x (-1.08), not a bare 1. Every mislabel that evening was this same collapse,
a LOCAL rise dressed up as the whole road. A slope's name must say both of its
ends -- "Q0 per b20" and "wrongness per Q0" are different animals, and only
their product is "wrongness per b20."
"The pull-sheet duplicates my numbers -- why does x show up in a whole column?"
The outer product looked like a copying bug until I counted what it builds: a
SHEET of dials needs a SHEET of pulls, one per dial. Input times signal builds it.
With x = [2.5, 0, 0] and seed [1, 0, 0, 0], every (input, signal) pair gets
multiplied and only C[0,0]'s pair is non-zero: 2.5 x 1 = 2.5. No copying -- twelve
small multiplications, eleven of them zero. One dial, one pull, always; a sheet's
pulls just come out sheet-shaped.
"Every new symbol felt invented -- grads, W, 0, 1 -- I could not say how many
things I was even tuning." The cure was to count the dials FIRST: sheet A has
2 x 3 = 6, its bias 3, sheet C has 3 x 4 = 12, its bias 4 -- 25 dials, full stop.
Every symbol in this post is one of those 25, or one of the fixed rows (s, psi, x,
Q) the forward pass makes. Before learning any machine, count its movable parts;
then no symbol can be a stranger.
SEAM. Pencil ends here; below, the same numbers in Python.
size = 0.01
# dials
s1, s2 = 1, 2
A00,A01,A02 = 1.0,-2.0, 0.5
A10,A11,A12 = 0.5, 1.0,-1.0
b10,b11,b12 = 0.5,-0.5, 1.0
C00,C10,C20 = 2.0, 1.0,-1.0 # column 0 (engine 0)
C03,C13,C23 = 1.0, 0.0,-1.0 # column 3 (engine 3)
b20,b23 = 0.5, -0.5
# forward pass
p1 = s1*A00 + s2*A10 + b10 # 1+1+0.5 = 2.5
p2 = s1*A01 + s2*A11 + b11 # -2+2-0.5 = -0.5
p3 = s1*A02 + s2*A12 + b12 # 0.5-2+1.0 = -0.5
x1 = p1 if p1 > 0 else 0 # 2.5
x2 = p2 if p2 > 0 else 0 # 0 (killed)
x3 = p3 if p3 > 0 else 0 # 0 (killed)
Q0 = x1*C00 + x2*C10 + x3*C20 + b20 # 2.5*2+0+0+0.5 = 5.5
Q3 = x1*C03 + x2*C13 + x3*C23 + b23 # 2.5*1+0+0-0.5 = 2.0
print(p1, p2, p3) # 2.5 -0.5 -0.5
print(x1, x2, x3) # 2.5 0 0
print(Q0, Q3) # 5.5 2.0
# backward walk: fired engine 0, how_wrong = 4.42 - 5.5 = -1.08
how_wrong = 4.42 - 5.5 # -1.08
pull_x1 = C00 # 2 (column 0 of C)
gate1 = 1 if p1 > 0 else 0 # 1 (p1=2.5 positive, gate open)
gate2 = 1 if p2 > 0 else 0 # 0 (p2 negative, gate shut)
gate3 = 1 if p3 > 0 else 0 # 0 (p3 negative, gate shut)
pull_psi1 = pull_x1 * gate1 # 2*1 = 2
pull_psi2 = C10 * gate2 # 1*0 = 0
pull_psi3 = C20 * gate3 # -1*0 = 0
pull_A00 = s1 * pull_psi1 # 1*2 = 2
pull_A10 = s2 * pull_psi1 # 2*2 = 4
print(pull_psi1, pull_psi2, pull_psi3) # 2 0 0
print(pull_A00, pull_A10) # 2 4
# wiggle test A[1,0]: claimed pull = 4
p1_w = s1*A00 + s2*0.501 + b10 # 1+1.002+0.5 = 2.502
Q0_w = p1_w*C00 + b20 # 2.502*2+0.5 = 5.504
print(Q0_w, (Q0_w-Q0)/0.001) # 5.504 4.0 -- pull confirmed
# wiggle test C[0,0]: claimed pull = 2.5
Q0_w2 = x1*2.001 + b20 # 2.5*2.001+0.5 = 5.5025
print(Q0_w2, (Q0_w2-Q0)/0.001) # 5.5025 2.5 -- pull confirmed
# spend the miss
A00_new = A00 + size*how_wrong*pull_A00 # 1.0+0.01*(-1.08)*2 = 0.9784
A10_new = A10 + size*how_wrong*pull_A10 # 0.5+0.01*(-1.08)*4 = 0.4568
b10_new = b10 + size*how_wrong*pull_psi1 # 0.5+0.01*(-1.08)*2 = 0.4784
C00_new = C00 + size*how_wrong*x1 # 2.0+0.01*(-1.08)*2.5 = 1.973
b20_new = b20 + size*how_wrong*1 # 0.5+0.01*(-1.08)*1 = 0.4892
print(round(A00_new,4), round(A10_new,4), round(b10_new,4)) # 0.9784 0.4568 0.4784
print(round(C00_new,4), round(b20_new,4)) # 1.973 0.4892
# post-nudge forward pass
p1_new = s1*A00_new + s2*A10_new + b10_new # 0.9784+2*0.4568+0.4784 = 2.3704
Q0_new = p1_new*C00_new + b20_new # 2.3704*1.973+0.4892 = 5.166
print(round(p1_new,4), round(Q0_new,3)) # 2.3704 5.166
# YOUR TURN: engine 3 backward walk (same gate, different column)
pull_psi1b = C03 * gate1 # 1*1 = 1
pull_A00b = s1 * pull_psi1b # 1*1 = 1
pull_A10b = s2 * pull_psi1b # 2*1 = 2
print(pull_psi1b, pull_A00b, pull_A10b) # 1 1 2
# THE FOLD: aim the wiggle ruler at psi = 0 exactly, from both sides
up = (max(0, 0.0 + 0.001) - max(0, 0.0)) / 0.001 # (0.001 - 0)/0.001 = 1.0
down = (max(0, 0.0 - 0.001) - max(0, 0.0)) / -0.001 # (0 - 0)/-0.001 = 0.0
print(up, down + 0.0) # 1.0 0.0 -- two probes, two answers: no one slope exists
gate_at_fold = 1 if 0.0 > 0 else 0
print(gate_at_fold) # 0 -- this code's own legislated choice: strictly-positive opens
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 15 -- Training the Q-Network):
Part 1 (this post) .
Part 2 -- A Smarter Step: Adam and Replay by Pencil .
Part 3 -- The Frozen Twin: A Batch of Misses at Once .
Part 4 -- The World Calls Three Times: Wiring the Agent by Pencil .
Part 5 -- From Eight Dials to a Soft Landing: The Whole Agent by Pencil
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================