==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
SPECIAL -- A NOTE THAT KILLS REPEATED WORK
What a Machine Keeps So It Stops Redoing Old Work
============================================================================================
A machine that writes text emits one word per turn, appends it, guesses again from a longer
line. Below, every operation is forced by a wall a turn earlier hits, and every matrix was
drawn fresh from a running program -- no number repeats across sections except where a repeat
IS a point being proven. Six numbers per word, three words, so a pencil checks any product. A
shipped machine runs wider over thousands of words and keeps an identical note.
--------
A MACHINE EMITS ONE WORD PER TURN, SO A LINE GROWS BY ONE
turn 1 [ word1 ]
turn 2 [ word1 word2 ]
turn 3 [ word1 word2 word3 ]
Wall: a guess yields one word, not a paragraph. To lengthen text a machine appends a guess and
re-guesses from a now-longer line. A line gains one word each turn.
--------
A NEWEST WORD CANNOT GUESS ITS FOLLOWER FROM ITSELF ALONE, SO IT READS EVERY EARLIER WORD
[ word1 word2 word3 ]
|___ reads back over word1, word2, word3 to guess word4
Wall: what follows "river" differs from what follows "money", so a follower depends on words
ahead of a newest word, not on a newest word by itself. A newest word reads across every
earlier word. Reading needs a number pulled from each earlier word -- forced next.
--------
READING A WORD MEANS RUNNING ITS SIX NUMBERS THROUGH A FIXED MATRIX, YIELDING A LABEL-ROW
three words, each a fresh row of six numbers:
word1 = [ -0.58 -0.86 -0.07 0.40 -0.14 -0.03 ]
word2 = [ -0.88 -0.02 0.80 0.63 0.41 -0.70 ]
word3 = [ 0.71 0.64 -0.60 0.24 -0.86 -0.69 ]
label-matrix Wk : six rows, six columns (drawn fresh)
[ -0.33 -0.62 0.47 0.57 -0.28 -0.33 ]
[ -0.70 -0.75 0.38 0.18 -0.80 -0.04 ]
[ -0.18 0.63 0.39 0.18 0.09 0.81 ]
[ 0.88 -0.29 -0.47 0.53 -0.79 -0.24 ]
[ -0.77 -0.33 -0.77 -0.38 0.52 0.73 ]
[ 0.53 0.11 0.21 -0.25 -0.60 -0.11 ]
label(word1) = word1 Wk = [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ]
label(word2) = word2 Wk = [ 0.03 0.67 -0.87 -0.01 0.47 1.16 ]
label(word3) = word3 Wk = [ -0.07 -1.16 0.75 1.04 -0.99 -1.36 ]
one entry by hand: label(word1)[0] = word1 . Wk[:,0]
= (-0.58)(-0.33)+(-0.86)(-0.70)+(-0.07)(-0.18)+0.40(0.88)+(-0.14)(-0.77)+(-0.03)(0.53)
= 1.25
Wall: a word arrives as six raw numbers, and raw numbers do not state what a word offers to
followers. Multiply a word-row by a fixed matrix Wk to cast a label-row -- six numbers standing
for what a word offers. To read an earlier word, a machine wants that word's label-row in hand.
--------
A LABEL-ROW DRAWS ON ONE WORD, SO NOTHING LATER CAN MOVE IT
word1 = [ -0.58 -0.86 -0.07 0.40 -0.14 -0.03 ] --Wk--> label(word1) = [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ]
^ only word1's six numbers feed this product; word2 and word3 never enter it
Wall: label(word1) is word1's numbers times Wk, and word2, word3 sit nowhere in that product.
So once word1 is emitted, label(word1) = [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ] is settled and
cannot move, no matter how long a line grows after.
--------
A MACHINE KEEPS NOTHING BETWEEN TURNS, SO EVERY TURN REBUILDS A LABEL-ROW FOR EVERY WORD
turn 1 [ word1 ] build label for: word1
turn 2 [ word1 word2 ] build label for: word1, word2
turn 3 [ word1 word2 word3 ] build label for: word1, word2, word3
Wall: between turns a machine holds no memory; each turn it receives a whole line and casts a
label-row for every word from a blank start. A longer line forces it to rebuild label-rows it
already built earlier.
--------
WORD1'S LABEL-ROW COMES OUT IDENTICAL AT TURNS 1, 2, 3, SO TWO OF THREE BUILDS ARE PURE WASTE
word1 = [ -0.58 -0.86 -0.07 0.40 -0.14 -0.03 ] , Wk fixed
turn 1 : word1 Wk = [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ]
turn 2 : word1 Wk = [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ]
turn 3 : word1 Wk = [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ]
no digit differs across three turns
Wall: word1's six numbers never move and Wk never moves, so word1's label-row prints identical
at turn 1, turn 2, turn 3. Three builds of one unchanging row; two arithmetic runs thrown out.
--------
BUILDS PILE UP AS 1+2+3, YET ONLY THREE LABEL-ROWS EVER EXIST
turn 1 builds 1 turn 2 builds 2 turn 3 builds 3
total built = 1 + 2 + 3 = 6 distinct rows = 3 wasted = 3
Wall: three words hold three label-rows, yet a no-memory machine built six. Word1's row built
thrice (two wasted), word2's twice (one wasted), word3's once. For a five-hundred-word line a
machine builds 1 + 2 + ... + 500 = 125,250 label-rows where only 500 exist. (One turn casts all
of a line's label-rows in a single matrix product, no loop; waste piles up ACROSS turns, each
turn redoing rows a prior turn already cast.)
--------
A ROW THAT NEVER MOVES SHOULD BE WRITTEN ONCE AND READ BACK, SO KEEP A NOTE
turn 1 : build label(word1) [1.25 0.89 -0.71 -0.23 0.47 -0.03] -> WRITE ON A NOTE
turn 2 : READ label(word1) off note ; build label(word2) ; ADD to note
turn 3 : READ label(word1), label(word2) off note ; build label(word3) ; ADD to note
Wall: rebuilding an unmoving row wastes arithmetic. Write a label-row on a note first time out,
read it back every turn after. Only a brand-new word needs a fresh label-row. A note holding
label-rows as a line grows is a cache.
--------
WITH A NOTE, A LABEL-ROW IS BUILT ONCE, SO 1+2+3 COLLAPSES TO 3
turn 1 builds 1 (word1) turn 2 builds 1 (word2) turn 3 builds 1 (word3)
total built = 1 + 1 + 1 = 3
Wall: read old rows off a note, cast only a newest row. Six drops to three. A five-hundred-word
line drops 125,250 builds to 500 -- a machine trades a pile that grows with a square of length
for one that grows straight.
--------
A LABEL-ROW CANNOT BE HANDED OVER, SO A SECOND MATRIX CASTS A PAYLOAD-ROW, KEPT ALONGSIDE
payload-matrix Wv : six rows, six columns (drawn fresh)
[ 0.42 -0.79 -0.86 0.49 -0.36 0.36 ]
[ 0.42 0.78 -0.18 -0.25 0.55 0.48 ]
[ 0.27 0.56 0.26 0.82 -0.30 0.43 ]
[ 0.81 -0.30 0.20 -0.24 -0.07 -0.76 ]
[ -0.87 0.47 -0.09 0.22 0.43 -0.15 ]
[ -0.24 -0.60 0.53 0.60 0.42 0.68 ]
payload(word1) = word1 Wv = [ -0.17 -0.42 0.71 -0.27 -0.34 -0.96 ]
payload(word2) = word2 Wv = [ 0.16 1.55 0.69 -0.25 -0.10 -1.00 ]
payload(word3) = word3 Wv = [ 1.51 -0.46 -1.12 -0.96 -0.40 -0.22 ]
Wall: a label-row rates fit, yet a reader also needs what a word hands over once leaned on. A
second fixed matrix Wv casts a payload-row from a word alone. Drawn from one word, a payload-row
never moves either, so it joins a note. A note now holds two rows per word: a label-row and a
payload-row.
--------
A NEWEST WORD ASKS ONCE, SO A WANT-ROW IS CAST FRESH AND NEVER KEPT
want-matrix Wq : six rows, six columns (drawn fresh)
[ 0.27 0.35 -0.55 0.48 -0.30 -0.11 ]
[ -0.33 0.12 0.29 0.14 -0.33 -0.51 ]
[ 0.62 -0.36 -0.11 -0.74 -0.36 -0.75 ]
[ -0.28 0.38 0.85 0.23 -0.42 0.28 ]
[ 0.23 0.59 0.06 0.74 -0.15 -0.39 ]
[ 0.03 0.76 0.40 -0.01 -0.00 0.27 ]
want(word3, newest) = word3 Wq = [ -0.68 -0.40 -0.26 0.30 -0.18 0.26 ]
match want(word3) against every label-row on a note, over root(6):
want(word3) . label(word1) / 2.45 = -0.48
want(word3) . label(word2) / 2.45 = 0.07
want(word3) . label(word3) / 2.45 = 0.18
Wall: a newest word needs a want-row -- what it hunts -- cast by a third matrix Wq, matched
against label-rows on a note to grade fit. Yet a want-row serves only on a turn where a word is
newest; next turn a word is old, and only its label-row and payload-row are read back. A spent
want-row is never wanted again, so it never joins a note. A note keeps label-rows and payload-
rows; want-rows are cast fresh for a newest word and dropped. Note also: Wk, Wv, Wq all read
one identical word-row -- three roles, one source.
--------
A SCORE SWINGS NEGATIVE AND GRADES ONLY FIT, SO EXPONENTIATE, SHARE OUT TO SUM 1, AND BLEND PAYLOAD-ROWS
three match scores for word3, one per word on a note: -0.48 0.07 0.18
force each positive, keeping order (e raised to a score):
e^(-0.48) = 0.62 e^(0.07) = 1.07 e^(0.18) = 1.20
total = 0.62 + 1.07 + 1.20 = 2.89
share out so all three sum to 1:
0.62 / 2.89 = 0.21 1.07 / 2.89 = 0.37 1.20 / 2.89 = 0.42
blend payload-rows off a note with those shares:
0.21 x [ -0.17 -0.42 0.71 -0.27 -0.34 -0.96 ] (payload word1)
+ 0.37 x [ 0.16 1.55 0.69 -0.25 -0.10 -1.00 ] (payload word2)
+ 0.42 x [ 1.51 -0.46 -1.12 -0.96 -0.40 -0.22 ] (payload word3)
= [ 0.65 0.29 -0.06 -0.55 -0.28 -0.66 ]
one entry by hand (column two):
0.21(-0.42) + 0.37(1.55) + 0.42(-0.46) = -0.09 + 0.57 - 0.19 = 0.29
Wall: a match score grades fit yet reads nothing back, and a raw score swings negative, so it
cannot serve as a mixing weight. Raising e to each score forces every value positive while
keeping order; dividing each by a total of all three makes shares sum to 1. Weighting payload-
rows (a payload-row: six numbers a word hands over once leaned on) by those shares folds all
three into one six-number row -- content drawn mostly from word3 (share 0.42) and word2 (share
0.37), a sliver from word1 (share 0.21) -- so a note never hands back one raw payload-row; it
feeds a blend. One last fixed matrix mixes that blended row before it joins a newest word's
running numbers; cast from no old word, that matrix never joins a note. (Exponentiate-then-
share carries a textbook name: softmax.)
--------
A MACHINE STACKS ROOMS, EACH WITH A DIFFERENT LABEL-MATRIX, SO EACH ROOM KEEPS A SEPARATE NOTE
room 1 label-matrix Wk -> label(word1) = [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ]
room 2 label-matrix Wk2 : six rows, six columns (drawn fresh)
[ -0.09 0.09 0.85 0.78 0.61 -0.67 ]
[ 0.10 -0.24 0.74 -0.34 0.78 -0.47 ]
[ -0.14 0.83 0.17 0.78 0.07 -0.49 ]
[ 0.38 0.04 0.01 -0.65 -0.64 0.77 ]
[ 0.82 0.84 -0.26 -0.82 0.10 0.43 ]
[ 0.27 -0.80 -0.48 0.37 -0.80 0.62 ]
room 2 label(word1) = word1 Wk2 = [ 0.01 0.02 -1.09 -0.37 -1.28 1.06 ]
Wall: a machine runs label / payload / want machinery many rounds -- call each round a room --
each room holding fresh matrices. Word1's label-row in room 1, [ 1.25 0.89 -0.71 -0.23 0.47
-0.03 ], differs sharply from word1's label-row in room 2, [ 0.01 0.02 -1.09 -0.37 -1.28 1.06 ].
So a note is per-room; a machine of twelve rooms keeps twelve separate notes, each holding
label-rows and payload-rows for that room.
In a shipped stack rooms chain: word1's six numbers entering room 2 are room 1's OUTPUT for
word1, not raw word1 (this page fed raw word1 into room 2 to keep pencil work light). That
output stays frozen too, for one reason -- inside every room word1 reads only itself and words
before it, never word2 or word3, so nothing arriving later can reach back and move word1's row
in any room. (This only-look-backward rule carries a textbook name: a causal mask; scores
toward later words are forced to minus infinity before sharing out, so those shares land on 0.)
Without that rule, every note up a stack would go stale each turn.
--------
PROOF -- RUN IT, GET THESE NUMBERS
Three words, fresh matrices, word1's label-row cast at three turns; a count of builds with a
note and without; a payload-row; want-matches; softmax shares and a blended payload-row; a
second room:
```python
import numpy as np, math
np.random.seed(11)
R = lambda *s: np.round(np.random.uniform(-0.9, 0.9, s), 2)
p = lambda v: [round(float(z), 2) for z in np.ravel(v)]
D = 6
W = R(3, D) # three words, six numbers each
Wk = R(D, D) # label-matrix
lab = lambda w: p(w @ Wk)
print("word1 label-row @ turn 1 :", lab(W[0]))
print("word1 label-row @ turn 2 :", lab(W[0]))
print("word1 label-row @ turn 3 :", lab(W[0]))
print("no digit differs? ", lab(W[0]) == lab(W[0]) == lab(W[0]))
print("builds without a note 1+2+3 =", 1+2+3, " with a note =", 3)
print("500 words without =", sum(range(1,501)), " with =", 500)
Wv = R(D, D) # payload-matrix
print("word1 payload-row :", p(W[0] @ Wv))
Wq = R(D, D) # want-matrix
want3 = W[2] @ Wq
for i in range(3):
print(f"want(word3).label(word{i+1})/root6 =", round(float(want3 @ (W[i] @ Wk)) / math.sqrt(D), 2))
e1 = math.exp(float(want3 @ (W[0] @ Wk)) / math.sqrt(D)) # 0.62
e2 = math.exp(float(want3 @ (W[1] @ Wk)) / math.sqrt(D)) # 1.07
e3 = math.exp(float(want3 @ (W[2] @ Wk)) / math.sqrt(D)) # 1.20
tot = e1 + e2 + e3 # 2.89
print("softmax shares :", round(e1/tot, 2), round(e2/tot, 2), round(e3/tot, 2))
blend = (e1/tot)*(W[0] @ Wv) + (e2/tot)*(W[1] @ Wv) + (e3/tot)*(W[2] @ Wv)
print("blended payload-row :", p(blend))
Wk2 = R(D, D) # a second room, fresh label-matrix
print("room1 word1 label-row :", lab(W[0]))
print("room2 word1 label-row :", p(W[0] @ Wk2))
```
Running a program prints:
```
word1 label-row @ turn 1 : [1.25, 0.89, -0.71, -0.23, 0.47, -0.03]
word1 label-row @ turn 2 : [1.25, 0.89, -0.71, -0.23, 0.47, -0.03]
word1 label-row @ turn 3 : [1.25, 0.89, -0.71, -0.23, 0.47, -0.03]
no digit differs? True
builds without a note 1+2+3 = 6 with a note = 3
500 words without = 125250 with = 500
word1 payload-row : [-0.17, -0.42, 0.71, -0.27, -0.34, -0.96]
want(word3).label(word1)/root6 = -0.48
want(word3).label(word2)/root6 = 0.07
want(word3).label(word3)/root6 = 0.18
softmax shares : 0.21 0.37 0.42
blended payload-row : [0.65, 0.29, -0.06, -0.55, -0.28, -0.66]
room1 word1 label-row : [1.25, 0.89, -0.71, -0.23, 0.47, -0.03]
room2 word1 label-row : [0.01, 0.02, -1.09, -0.37, -1.28, 1.06]
```
Word1's label-row prints [ 1.25 0.89 -0.71 -0.23 0.47 -0.03 ] at all three turns; six builds
where three suffice; a room 2 label-matrix casts a fully different row. A note that writes a
label-row and a payload-row once, then reads back, is a KV cache.
--------
>> NOTE: STANDARD JARGON
Plain term above textbook name
row of six numbers a vector
label-row key (K)
payload-row value (V)
want-row query (Q)
a note a KV cache
write once, read back store keys and values, reuse across turns
one word per turn autoregressive generation
a turn a decoding pass
a room a transformer block (layer)
exponentiate, share to 1 softmax (attention weights)
a blended payload-row attention output
final mixing matrix output projection (W_O)
only-look-backward rule causal mask
Two scale notes for a shipped machine. First: each room splits width 768 into twelve lanes of
64 numbers (textbook: attention heads); every lane runs match-and-blend on 64-wide rows and
keeps a slice of a note, so a match divides by root(64) = 8 -- never by root of full width.
Second: after blending, each room also runs a widen-bend-shrink stage (textbook: MLP) on a
newest word alone; it reads no old word, so a note holds nothing for it.
--------
A LADDER RUNG -- WHAT FORCED THIS PAGE, AND WHAT IT FORCES NEXT
Rungs below: a smallest writing machine of one room
(A Writing Machine of One Room), then twelve rooms stacked
(What a Machine Does When You Press "h") -- and both
rebuild every old row every turn, a waste this page's note kills. A wall remains: a note
grows with a line, and a full score grid grows with a square of length -- capping each
word's reach is Sliding Window Attention by Pencil.
A whole ladder, rung by rung, sits at Build a GPT, Forced.