==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 22 . THE FATE OF EVERY NUMBER . PART 1 OF 3
The 90 Comes Back As 89.94
============================================================================================
Every textbook shows the fate of variables: W goes in, dW comes out. None of
them follows one plain number -- where the 90 sitting in a dial actually
goes, which other numbers it gets fused into, how far its push reaches into
the loss, and what exact arithmetic reaches back and turns it into 89.94.
This chapter follows single numbers, one at a time, forward into the loss
and backward to the dial. No number gets to hide inside a symbol.
The machine first, because every fate starts at the same place. Twelve
numbers are dials -- the machine may turn them. Eight numbers arrive from
outside and may not be touched: four inputs, four targets.
inputs 2 1 targets 100 55
1 3 117 33
first room dials 3 -2 free adds 2 -8
1 5
second room dials 2 1 free adds 90 40
3 -1
The machine's whole forward run is three moves: multiply-and-add through
the first room, bend every negative up to zero, multiply-and-add through
the second room. Then one loss number. Every multiplication written out:
first room, multiply then add the free adds:
2*3 + 1*1 = 7 7 + 2 = 9
2*(-2) + 1*5 = 1 1 + (-8) = -7
1*3 + 3*1 = 6 6 + 2 = 8
1*(-2) + 3*5 = 13 13 + (-8) = 5
the bend (negatives become 0, positives pass):
max(9, 0) = 9
max(-7, 0) = 0
max(8, 0) = 8
max(5, 0) = 5
second room, multiply then add the free adds:
9*2 + 0*3 = 18 18 + 90 = 108
9*1 + 0*(-1) = 9 9 + 40 = 49
8*2 + 5*3 = 31 31 + 90 = 121
8*1 + 5*(-1) = 3 3 + 40 = 43
the four guesses against the four targets:
108 - 100 = 8 8*8 = 64
49 - 55 = -6 (-6)*(-6) = 36
121 - 117 = 4 4*4 = 16
43 - 33 = 10 10*10 = 100
64 + 36 + 16 + 100 = 216
216 / 4 = 54
The machine's guesses miss by 8, -6, 4, and 10. Squared and averaged, the
whole run is worth one number: the loss is 54. Everything below is about
making that 54 smaller -- and the only things the machine may move are its
twelve dials.
-------
THE FATE OF 90
The 90 is one of the free adds in the second room. Follow it and nothing
else. It sits in its slot doing nothing while the first room runs (7, 1, 6,
13 get built, the free adds 2 and -8 turn them into 9, -7, 8, 5, the bend
turns those into 9, 0, 8, 5) and while the second room multiplies (18 and
31 get built on the first output, 9 and 3 on the second). Then its moment:
the + hands a copy of 90 to the 18 and a second copy of 90 to the 31.
18 + 90 = 108
31 + 90 = 121
Two copies did work. And here is the uncomfortable part: inside 108, the 90
is gone. Nobody looking at 108 can split it back into 18 and 90 -- the +
fused them for good. Same for 121. What survives is the slot: the dial
still holds 90, untouched, because the forward run only read it.
The two fused copies keep riding:
108 - 100 = 8 8*8 = 64
121 - 117 = 4 4*4 = 16
and both squares land inside 216, which becomes 54. So the 90 sits inside
the loss twice -- once through 108, once through 121. It never touched 49
or 43; those got the other free add, the 40.
Anyone claiming the 90 "does nothing because it was just added" can be
tested. Put 91 in the slot and run the whole machine again. The first room
and the bend do not contain the slot, so 9, 0, 8, 5 come out unchanged; the
second room's multiplies are unchanged too (18, 9, 31, 3). Only the two
stamped copies differ:
18 + 91 = 109 109 - 100 = 9 9*9 = 81
31 + 91 = 122 122 - 117 = 5 5*5 = 25
81 + 36 + 25 + 100 = 242
242 / 4 = 60.5
One unit into the slot, and the loss jumped from 54 to 60.5. The 90 is
wired straight into the loss, through both of its copies.
-------
THE ARITHMETIC THAT REACHES BACK
Now the direction nobody draws: from the 54, back to the slot. The loss
needs no new machinery -- it is the four squares averaged:
54 = (64 + 36 + 16 + 100) / 4
= ((108-100)^2 + (49-55)^2 + (121-117)^2 + (43-33)^2) / 4
First question: if 108 alone moved a tiny amount h, how much does 54 move?
Only the first square contains 108:
(108 + h - 100)^2 = (8 + h)^2 = 64 + 16h + h^2
so the loss moves by (16h + h^2)/4. For tiny h the h^2 part is nothing, so
the move per unit is 16/4 = 4. The slope of the loss at 108 is 4. Same
question at 121, where the third square lives:
(121 + h - 117)^2 = (4 + h)^2 = 16 + 8h + h^2
slope 8/4 = 2. (This also settles the whole-unit test, where 91 in the slot
drove the loss to 60.5: the slopes promise 4 + 2 = 6 per unit, the test
measured 6.5, and the extra 0.5 is exactly the two h^2 terms at h = 1:
(1 + 1)/4. Slopes are read standing still; walk a whole unit and the
square's own bend adds on.)
Second question: if the slot moved by h, how much do 108 and 121 move?
108 = 18 + 90, and the 18 has no 90 in it, so 108 moves by exactly h --
the + passes the wiggle through at full size, one for one. Same for 121.
This is the answer to "we cannot un-fuse the 90 out of 108": nobody needs
to. The value fused; the wiggle still passes through at rate 1.
So one wiggle of the slot moves the loss along two roads at once:
through 108: 4 per unit * 1 = 4
through 121: 2 per unit * 1 = 2
Both pushes land on the same single slot -- one number cannot feel them
separately, and the loss is itself a plus of four squares, so the two
arrivals add:
4 + 2 = 6
Move the slot up by one unit, the loss rises by 6 (plus bend). Therefore
move it DOWN, and only a little -- one hundredth of the pull:
90 - 0.01 * 6 = 90 - 0.06 = 89.94
The slot is overwritten. The 90 that was copied, fused, and lost inside
108 and 121 was never recovered -- it never had to be. The original never
left home, and the arithmetic that came back for it needed only the two
slopes (4 and 2), the two pass-through rates (1 and 1), and one add.
-------
THE FATE OF 40, WHERE THE TWO ROADS FIGHT
Same machine, from the top, other free add. The first room builds 7, 1, 6,
13; the free adds 2 and -8 make 9, -7, 8, 5; the bend makes 9, 0, 8, 5;
the second room multiplies out 18, 9, 31, 3. The + hands one copy of 40 to
the 9 and one copy to the 3:
9 + 40 = 49
3 + 40 = 43
Two copies, fused, riding on:
49 - 55 = -6 (-6)*(-6) = 36
43 - 33 = 10 10*10 = 100
and both squares sit inside 216/4 = 54. The 40 never touched 108 or 121.
Backward. Slope of the loss at 49 -- only the second square holds 49:
(49 + h - 55)^2 = (-6 + h)^2 = 36 - 12h + h^2
slope -12/4 = -3. Negative: pushing 49 UP makes the loss FALL, because 49
sits six below its target of 55. Slope at 43:
(43 + h - 33)^2 = (10 + h)^2 = 100 + 20h + h^2
slope 20/4 = 5. Positive: 43 sits ten above its target of 33, pushing it
up makes things worse. The + passes the slot's wiggle through at rate 1
into both, so the two roads report back:
through 49: -3 * 1 = -3 (this road wants the 40 BIGGER)
through 43: 5 * 1 = 5 (this road wants the 40 SMALLER)
The two copies of 40 disagree -- one of its guesses came in under its
target, the other came in over. But there is one slot, so there is one
verdict, and it is the add:
-3 + 5 = 2
The over-shooting road pulls harder than the under-shooting road, so the
net says: down, a little.
40 - 0.01 * 2 = 40 - 0.02 = 39.98
And the whole-unit test agrees: put 41 in the slot, the copies build 50
and 44, the misses become -5 and 11, the squares 25 and 121, and the loss
is (64 + 25 + 16 + 121)/4 = 226/4 = 56.5 -- up by 2.5, which is the slope
2 plus the same 0.5 of bend as before.
-------
Both fates, run as code -- no shortcuts, the same arithmetic:
```python
# forward: first room, bend, second room -- pure arithmetic
r1 = 2*3 + 1*1 + 2 # 9
r2 = 2*(-2) + 1*5 + (-8) # -7
r3 = 1*3 + 3*1 + 2 # 8
r4 = 1*(-2) + 3*5 + (-8) # 5
b1_ = max(r1, 0) # 9
b2_ = max(r2, 0) # 0 <- the -7 dies at the bend
b3_ = max(r3, 0) # 8
b4_ = max(r4, 0) # 5
g1 = b1_*2 + b2_*3 + 90 # 108
g2 = b1_*1 + b2_*(-1) + 40 # 49
g3 = b3_*2 + b4_*3 + 90 # 121
g4 = b3_*1 + b4_*(-1) + 40 # 43
loss = ((g1-100)**2 + (g2-55)**2 + (g3-117)**2 + (g4-33)**2) / 4
print("loss:", loss)
# fate test: 90 -> 91, rerun only what changes
loss_91 = ((109-100)**2 + (49-55)**2 + (122-117)**2 + (43-33)**2) / 4
print("loss with 91:", loss_91)
# backward for the 90: two slopes, two rate-1 roads, one add
slope_108 = 2*(108-100)/4 # 4.0
slope_121 = 2*(121-117)/4 # 2.0
pull_90 = slope_108*1 + slope_121*1
print("pull on 90:", pull_90)
print("90 becomes:", 90 - 0.01*pull_90)
# backward for the 40: the two roads disagree
slope_49 = 2*(49-55)/4 # -3.0
slope_43 = 2*(43-33)/4 # 5.0
pull_40 = slope_49*1 + slope_43*1
print("pull on 40:", pull_40)
print("40 becomes:", 40 - 0.01*pull_40)
```
Running this code prints:
loss: 54.0
loss with 91: 60.5
pull on 90: 6.0
90 becomes: 89.94
pull on 40: 2.0
40 becomes: 39.98
Two of the twelve dials have now made the full round trip: copied by the
+, fused into two guesses, carried into two squares, reached back by two
slopes times two rate-1 roads, added, and overwritten. Ten dials remain --
and the next ones are not handed to the + at rate 1. They get MULTIPLIED
on their way in, by 9, by 8, by 2, by 1, and their roads carry those
multipliers back. One of those hand-worked numbers, a 52, will force a
flip that no spoken rule can explain away. That is part 2.
-------
>> NOTE: STANDARD JARGON
the free add = the bias; the pair [90, 40] is a bias vector, and stamping it onto both lines is called broadcasting
the slope = the gradient (also: the derivative); slope of the loss at 108 = dL/d(guess)
the pull on a dial = that dial's entry in the gradient; two roads adding = the sum in db = dL_dguess.sum(axis=0)
the tweak = one SGD update: dial = dial - learning_rate * pull, here 90 - 0.01*6
the bend = ReLU, max(x, 0)