==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 15 . TRAINING THE Q-NETWORK . PART 2 OF 5
A Smarter Step: Adam and Replay by Pencil
============================================================================================
A dial = one learned number in a
network. A worth = the network's guess of how much reward a move still brings.
How-wrong = target minus guess, one number per graded move (say -1.08: the guess
was 1.08 too high). A dial's pull = how much raising that dial raises the graded
worth. The previous post nudged every dial by
dial = dial + size x (how wrong) x (its pull)
Call the signed product (how wrong) x (pull) the dial's GRADE for this move -- one
number per dial, already carrying its direction: a grade of +0.2 says "raise me a
little," -0.2 says "lower me." The nudges were right. The way we SPENT them was
crude, in two separate ways, and this post fixes both. The step: a flat size 0.01
for every dial, on every move, forever -- ignoring each dial's history and scale.
The data: each move's experience used ONCE, in the order it happened, then thrown
away. Fix the step first, then the data.
SO GIVE THE STEP A MEMORY: AVERAGE EACH DIAL'S RECENT GRADES
A flat step treats every grade as a surprise. But a dial graded the SAME way move
after move is clearly being told something steady -- it should pick up speed, not
inch. A dial graded one way then the other is getting mixed messages -- it should
hold still. So keep, per dial, a running average of its recent grades. Call it m
(for momentum):
m = beta1 x m + (1 - beta1) x grade (beta1 = 0.9: m is nine-tenths old, one-tenth new)
m is the dial's smoothed direction. A dial graded +0.2, +0.2, +0.2 builds an m
near +0.2 -- steady push. A dial graded +0.2, -0.2, +0.2 builds an m near 0 -- the
yanks cancel. Drawn:
grades in: +0.2 +0.2 +0.2 -> m climbs toward +0.2 (steady -> full speed)
grades in: +0.2 -0.2 +0.2 -> m hovers near 0 (mixed -> crawl)
WHICH STILL NEEDS SCALING, BECAUSE DIALS COME IN DIFFERENT SIZES
One dial's grades might run around 0.2; another's around 50. A single step size
cannot suit both -- 0.01 x 50 lurches, 0.01 x 0.2 dawdles. So keep a SECOND
running average per dial: the average of its SQUARED grades. Call it v:
v = beta2 x v + (1 - beta2) x grade^2 (beta2 = 0.999: v is a slow, long memory)
The square throws away the sign and keeps only the SIZE, so sqrt(v) is roughly how
big this dial's grades typically are. Divide the step by sqrt(v) and every dial
moves at the same measured pace, whatever its raw scale:
step for a dial ~ m / sqrt(v) (smoothed direction, divided by its typical size)
A big-grade dial and a small-grade dial both end up taking a step of about the
same length -- the size is normalized away, only the direction and steadiness
remain.
AND BECAUSE BOTH START AT ZERO, UNDO THE EARLY LOWBALL
m and v both start at 0 (no history yet). On the first few moves that 0 drags the
average DOWN -- after one grade of 0.2, m = 0.1 x 0.2 = 0.02, far below the real
0.2. How much of the zero-start is still inside the average after t moves? Exactly
beta^t of it. So divide by what remains:
m-hat = m / (1 - beta1^t) v-hat = v / (1 - beta2^t)
This bias-correction blows the early estimates back up to true size. The full
step, with a base size alpha and a hair epsilon so you never divide by zero:
dial = dial + alpha x m-hat / ( sqrt(v-hat) + epsilon )
That whole machine -- momentum m, scaler v, both bias-corrected -- is Adam. The
direction is already inside the grade (a negative grade makes m negative and the
step negative), so the update simply ADDS, exactly like the previous post's nudge.
ADAM BY HAND -- TWO DIALS, THE STEADY ONE AND THE FLIP-FLOP
alpha = 0.1, beta1 = 0.9, beta2 = 0.999, epsilon a hair (ignore it). Both dials
start with m = 0, v = 0. Watch a STEADY dial (grade +0.2 every move) beside a
FLIP-FLOP dial (+0.2 then -0.2).
MOVE 1 -- both get grade +0.2, so both do the identical thing:
m = 0.9 x 0 + 0.1 x 0.2 = 0.02
v = 0.999 x 0 + 0.001 x (0.2)^2 = 0.001 x 0.04 = 0.00004
m-hat = 0.02 / (1 - 0.9) = 0.02 / 0.1 = 0.2 (bias-corrected back to true 0.2)
v-hat = 0.00004 / (1 - 0.999) = 0.00004 / 0.001 = 0.04 -> sqrt(v-hat) = 0.2
step = alpha x m-hat / sqrt(v-hat) = 0.1 x 0.2 / 0.2 = 0.1
The first step is the full alpha = 0.1, and notice WHY: m-hat / sqrt(v-hat) =
0.2 / 0.2 = 1, the magnitude cancels and only the SIGN survives. Adam's first move
is alpha-sized whatever the grade's raw size -- more on that sting below.
MOVE 2 -- here the two dials part. The STEADY dial gets +0.2 again:
m = 0.9 x 0.02 + 0.1 x 0.2 = 0.018 + 0.02 = 0.038
m-hat = 0.038 / (1 - 0.81) = 0.038 / 0.19 = 0.2
v = 0.999 x 0.00004 + 0.001 x 0.04 = 0.00007996
v-hat = 0.00007996 / (1 - 0.998001) = 0.00007996 / 0.001999 = 0.04 -> sqrt = 0.2
step = 0.1 x 0.2 / 0.2 = 0.1 (full stride again)
The FLIP-FLOP dial gets -0.2 on move 2 instead:
m = 0.9 x 0.02 + 0.1 x (-0.2) = 0.018 - 0.02 = -0.002
m-hat = -0.002 / 0.19 = -0.0105 (to four decimals)
v = 0.00007996 (same -- squaring kills the sign) -> sqrt(v-hat) = 0.2
step = 0.1 x (-0.0105) / 0.2 = -0.005 (a twentieth of the steady stride)
Drawn, the two dials after move 2:
STEADY (+0.2, +0.2) : step 0.1 (m held its direction -> full speed)
FLIP-FLOP (+0.2, -0.2) : step -0.005 (m nearly cancelled -> near standstill)
Same base alpha, same scaler; the only difference is whether the grades AGREED.
Consistent grades compound into speed; contradictory grades cancel into caution.
That is the whole gift of Adam over the flat 0.01.
Move 3 of the STEADY dial (a third grade of +0.2 arrives; m = 0.038
and v = 0.00007996 from move 2 above). Compute m, m-hat, and the step.
CHECK: m = 0.9 x 0.038 + 0.1 x 0.2 = 0.0342 + 0.02 = 0.0542
m-hat = 0.0542 / (1 - 0.9^3) = 0.0542 / 0.271 = 0.2 -- exactly 0.2 again
v = 0.999 x 0.00007996 + 0.001 x 0.04 = 0.00011988 -> v-hat = 0.04, sqrt 0.2
step = 0.1 x 0.2 / 0.2 = 0.1
A perfectly steady dial strides the full alpha every single move -- the
bias correction keeps m-hat pinned at the true 0.2 while m itself is
still crawling up (0.02, 0.038, 0.0542, ...).
BUT THE STEP IS ONLY HALF THE WASTE -- THE OTHER HALF IS THE DATA
Adam fixed HOW we spend a grade. The second crude thing was the DATA: the net
learned from each move once, the instant it happened, in the order it happened.
Two costs.
First, a real run is a STREAK -- the lander drifts through a long stretch of
near-identical states, so the net sees the same kind of situation again and again
and forgets the ones from a minute ago (it overwrites its dials to fit the
streak). Second, one move is one lesson, then gone -- expensive footsteps, learned
from once.
WHICH IS WHY YOU KEEP A BIN OF PAST MOVES AND DRAW FROM IT AT RANDOM
Keep a bin -- a buffer -- of recent experiences. After every real move, drop in
one five-part note: the spot you left, the move you made, the reward, whether the
game ENDED on that move (1 yes / 0 no), and the spot you landed on.
one note : ( spot s , move a , reward r , ended? , landing s' )
the bin : [ note, note, note, ... ] newest pushed in, oldest dropped
(The ended? flag must ride in the note. A note is re-read long after the move
happened, and nothing else in it says whether a landing exists to peek at.)
Then, to LEARN, do not use the move you just made. Reach into the bin and pull out
a RANDOM handful -- a batch -- of old notes, mixed from all over the run, and take
one Adam step on their averaged grades:
real move -> drop the five-part note in the bin
to learn -> grab a random batch from the bin -> forward, target, backprop, Adam step
Two wins at once. The random grab MIXES far-apart moves into one batch, so no
single streak dominates -- the correlation is broken. And every note sits in the
bin to be drawn many times, so one costly footstep teaches over and over.
Each time an old note is drawn, its target y is rebuilt with the CURRENT net --
reward + dilute x (chance-weighted worths the net NOW reads at s'), and the whole
future piece times (1 - ended?), so a note that ended the game keeps only its
reward. The note is fixed; the net has moved since; so the target is fresh every
draw. (This is the notebook-and-rehearsal trick from the Dyna post, with one
change: the replayed move is a REAL stored transition, not one a learned model
dreamed up.)
One note, drawn twice. The note says (s, move 0, reward 1, ended? 0,
s'), dilute 0.9, chances at s' = [0.1, 0.6, 0.1, 0.2]. On the first draw the net
reads Q(s') = [3, 5, 1, 2]. By the second draw the net has learned a little and
reads Q(s') = [3, 4, 1, 2]. Compute both targets.
CHECK: draw 1: averaged = 0.1x3 + 0.6x5 + 0.1x1 + 0.2x2 = 3.8
y = 1 + 0.9 x 3.8 = 4.42
draw 2: averaged = 0.1x3 + 0.6x4 + 0.1x1 + 0.2x2 = 0.3+2.4+0.1+0.4 = 3.2
y = 1 + 0.9 x 3.2 = 3.88
Same note, two different targets -- the note never changes, the net's
read of the landing does. That is why the target is rebuilt per draw,
never stored.
SO THE WHOLE TRAINING LOOP, ONE REAL MOVE AT A TIME
Everything so far, in one beat-by-beat loop:
(1) READ the worths Q(s) from the net (forward: psi = sA+b1, x = max(psi,0), Q = xC+b2)
(2) PICK a move -- mostly the biggest worth, epsilon of the time a wander
(3) ACT, and the world hands back (reward, next spot s', ended?)
(4) STORE the five-part note (s, a, r, ended?, s') in the bin
(5) GRAB a random batch from the bin
(6) for each note in the batch: rebuild target y (future piece times 1 - ended?),
backprop the one-slot how-wrong, add up each dial's grade
(7) STEP every dial once with Adam, on the batch's averaged grades
(8) SLIDE: s' becomes the new s; repeat
Beat 1 and 2 are the Q-network post; beat 6's backprop is the previous post; beats
4-5 are the bin; beat 7 is Adam. That loop, run on the lander, is a complete
deep-RL agent -- the remaining posts of this chapter batch it, wire it to the
world's three calls, and run it to a landing.
TWO AVERAGES OF ONE GRADE, A HUGE FIRST STEP, AND OTHER MISREADINGS
"m and v are two averages of the same thing -- why keep both?" Feed the flip-flop
(+0.2, -0.2) into each and they split: m goes 0.02 then -0.002 -- it SEES the
disagreement and stalls. v goes 0.00004 then 0.00007996 -- identical to the steady
dial's v, because squaring erased every sign. v cannot see disagreement; m cannot
see scale. m answers "which way, and do the grades agree?"; v answers "how big are
they, usually?" Two questions, two memories.
"My first grade was huge, so the first step should be huge." Give one dial a first
grade of 0.2 and another 50, and watch. The small one: m-hat = 0.2, sqrt(v-hat) =
0.2, step = 0.1 x 1 = 0.1. The big one: m-hat = 50, v-hat = 2500, sqrt = 50, step
= 0.1 x 50/50 = 0.1. Identical. On move one, m-hat / sqrt(v-hat) is ALWAYS +1 or
-1 -- Adam is size-blind at birth and only learns a dial's scale as history
accumulates. Alpha alone sets the first stride; choose it as "the biggest step I
am happy to take blind."
"The bias correction looks like magic -- where does beta^t come from?" From one
honest fraction. After one move, m = 0.9 x 0 + 0.1 x 0.2 = 0.02 -- the average is
90% made of the fake zero start, and only 1 - 0.9^1 = 0.1 of it is real signal.
Divide by exactly that: 0.02 / 0.1 = 0.2, the true grade. After two moves the real
share is 1 - 0.9^2 = 0.19; check: 0.038 / 0.19 = 0.2 again. The correction divides
by the fraction of the average that is real. Nothing else.
"I stored the target y in the note -- it saves recomputing." The drill above is
the counterexample: the same note honestly graded 4.42 on one draw and 3.88 a
while later, because the net's read of the landing had moved. A stored 4.42 would
teach the old net's opinion forever, long after it stopped being held. A note
stores FACTS (s, a, r, ended?, s'). Opinions -- targets -- are rebuilt from the
current net at every draw.
SEAM. Pencil ends here; below, the same numbers in Python.
import math
alpha, beta1, beta2 = 0.1, 0.9, 0.999
# move 1: both dials get grade +0.2
m1 = 0.9*0 + 0.1*0.2 # 0.1*0.2 = 0.02
v1 = 0.999*0 + 0.001*0.2**2 # 0.001*0.04 = 0.00004
mh1 = m1 / (1 - 0.9**1) # 0.02/0.1 = 0.2
vh1 = v1 / (1 - 0.999**1) # 0.00004/0.001 = 0.04
step1 = alpha * mh1 / math.sqrt(vh1) # 0.1*0.2/0.2 = 0.1
print(round(mh1,4), round(vh1,4), round(step1,4)) # 0.2 0.04 0.1
# move 2: STEADY dial gets +0.2 again
m2s = 0.9*m1 + 0.1*0.2 # 0.018+0.02 = 0.038
v2s = 0.999*v1 + 0.001*0.04 # ~0.00007996
mh2s = m2s / (1 - 0.9**2) # 0.038/0.19 = 0.2
step2s = alpha * mh2s / math.sqrt(v2s/(1-0.999**2)) # 0.1*0.2/0.2 = 0.1
print(round(mh2s,4), round(step2s,4)) # 0.2 0.1
# move 2: FLIP-FLOP dial gets -0.2
m2f = 0.9*m1 + 0.1*(-0.2) # 0.018-0.02 = -0.002
mh2f = m2f / (1 - 0.9**2) # -0.002/0.19 = -0.01053
step2f = alpha * mh2f / math.sqrt(v2s/(1-0.999**2)) # ~-0.005 (a twentieth of 0.1)
print(round(m2f,5), round(mh2f,5), round(step2f,4)) # -0.002 -0.01053 -0.0053
# move 3: STEADY dial (m2s=0.038, v2s carried forward)
m3s = 0.9*m2s + 0.1*0.2 # 0.0342+0.02 = 0.0542
v3s = 0.999*v2s + 0.001*0.04 # ~0.00011988
mh3s = m3s / (1 - 0.9**3) # 0.0542/0.271 = 0.2
step3s = alpha * mh3s / math.sqrt(v3s/(1-0.999**3)) # 0.1*0.2/0.2 = 0.1
print(round(m3s,5), round(mh3s,4), round(step3s,4)) # 0.0542 0.2 0.1
# experience replay: note (s, move0, r=1, ended?=0, s'), dilute=0.9
# draw 1: Q(s')=[3,5,1,2], chances=[0.1,0.6,0.1,0.2]
avg1 = 0.1*3 + 0.6*5 + 0.1*1 + 0.2*2 # 0.3+3.0+0.1+0.4 = 3.8
y1 = 1 + 0.9*avg1 # 1+3.42 = 4.42
print(avg1, y1) # 3.8 4.42
# draw 2: net moved; Q(s') now [3,4,1,2]
avg2 = 0.1*3 + 0.6*4 + 0.1*1 + 0.2*2 # 0.3+2.4+0.1+0.4 = 3.2
y2 = 1 + 0.9*avg2 # 1+2.88 = 3.88
print(avg2, round(y2,4)) # 3.2 3.88
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 15 -- Training the Q-Network):
Part 1 -- Backpropagation by Pencil .
Part 2 (this post) .
Part 3 -- The Frozen Twin: A Batch of Misses at Once .
Part 4 -- The World Calls Three Times: Wiring the Agent by Pencil .
Part 5 -- From Eight Dials to a Soft Landing: The Whole Agent by Pencil
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================