==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
SPECIAL -- ONE KEYPRESS, EVERY MATRIX FORCED INTO BEING
What a Machine Does When You Press "h"
============================================================================================
Press "h". A machine ranks every mark it knows for what follows, and bets one. Below, each
operation is forced by a mechanical wall a round earlier hits -- no operation arrives before a
wall demands it. Every matrix shown was drawn fresh from a running program, so no number
repeats across sections; a person can redo any product with a pencil. Widths are six here, ten
marks, twelve rooms. A shipped machine runs six-hundred-plus wide over near fifty-thousand
marks, and walks an identical path.
--------
A KEY-MARK IS A SHAPE, NOT A NUMBER, SO PAIR EACH MARK WITH AN INTEGER ONCE
'a' -> 0 'b' -> 1 'c' -> 2 'd' -> 3
'e' -> 4 'f' -> 5 'g' -> 6 'h' -> 7
Wall: multiplication acts on numbers, and a mark drawn on a key is a shape, so "h" times
anything is undefined. A pairing of marks to integers, fixed once, breaks that wall: "h"
becomes integer 7. (A shipped machine pins "h" at 71 inside a near-fifty-thousand pairing; 7 is
a fact about a chosen pairing, not about "h".) From now, 7 is all a keypress leaves behind.
--------
AN INTEGER NAMES A SHELF BUT CARRIES NO TRAITS, SO PARK A ROW OF NUMBERS PER MARK
table E : ten marks down, six traits across
[ -0.76 0.50 -0.11 0.40 0.86 0.07 ] mark 0
[ 0.00 -0.77 -0.42 -0.00 0.32 0.55 ] mark 1
[ -0.21 -0.78 -0.38 0.74 -0.52 -0.09 ] mark 2
[ 0.78 -0.86 0.18 0.81 -0.49 0.09 ] mark 3
[ 0.74 -0.66 0.04 0.45 0.30 -0.06 ] mark 4
[ -0.53 -0.02 -0.23 -0.04 -0.24 0.61 ] mark 5
[ 0.48 -0.33 0.13 -0.40 -0.08 -0.26 ] mark 6
[ 0.28 -0.23 -0.07 0.39 -0.16 0.73 ] mark 7 <- "h"
[ -0.58 0.43 -0.14 -0.13 0.24 0.04 ] mark 8
[ -0.15 -0.90 -0.73 0.38 0.04 0.35 ] mark 9
read row 7: e = [ 0.28 -0.23 -0.07 0.39 -0.16 0.73 ]
Wall: 7 is a shelf label, and a label has no size, sign, or trait to grind into a guess. Park a
fresh row of six numbers under every mark; reading row 7 hands back six numbers a machine can
multiply. A row per mark, a shelf per integer: a lookup table. Row 7 speaks for "h" now.
--------
TWO IDENTICAL MARKS AT DIFFERENT SLOTS READ ONE ROW, YET ORDER FLIPS MEANING, SO ADD A SLOT-ROW
table P : eight slots down, six numbers across
[ 0.82 0.33 -0.80 -0.34 0.17 -0.48 ] slot 0
[ 0.84 0.80 0.63 -0.05 0.61 -0.66 ] slot 1
[ -0.34 -0.07 0.44 -0.03 -0.65 -0.28 ] slot 2
[ -0.32 -0.36 -0.60 -0.15 -0.09 0.49 ] slot 3
[ 0.53 0.04 -0.07 0.50 0.70 0.31 ] slot 4
[ 0.54 0.79 -0.83 0.68 -0.40 -0.04 ] slot 5
[ 0.53 0.39 -0.64 0.29 -0.78 -0.26 ] slot 6
[ 0.56 -0.13 0.18 0.41 0.58 0.47 ] slot 7
e = [ 0.28 -0.23 -0.07 0.39 -0.16 0.73 ] ("h", any slot)
+ slot 0 = [ 0.82 0.33 -0.80 -0.34 0.17 -0.48 ] (sits first)
-------------------------------------------------------
x = [ 1.10 0.10 -0.87 0.05 0.01 0.25 ]
Wall: "h" first and "h" fifth read one identical row, yet a guesser must part them -- order
bends meaning. A second table holds a fresh row per slot; adding slot 0's row onto e stamps
position into six numbers. Call a running six-number row x. Length six never budges downstream;
values do.
--------
ONE LOUD COORDINATE DROWNS A PRODUCT, SO CENTER TO ZERO AND DIVIDE BY SPREAD
u = [ -0.89 -0.14 -0.07 -0.80 0.07 0.19 ]
mean = ( -0.89 -0.14 -0.07 -0.80 +0.07 +0.19 ) / 6 = -0.27
spread = root of average squared gap from mean = 0.42
norm_i = ( u_i - (-0.27) ) / 0.42
norm = [ -1.47 0.32 0.49 -1.26 0.82 1.11 ]
check: mean(norm) = 0.00, spread(norm) = 1.00
Wall: a product row-times-matrix sums six terms, and one coordinate at 8.0 beside others near
0.1 hijacks every sum -- sign of a result turns hostage to a single loud entry. Center a copy to
mean 0, divide by spread, so no coordinate towers over rest. A tiny guard sits under a root
against a divide by zero. This flattened copy feeds forward; x waits untouched, to catch a
result added on. Name: layer normalization.
--------
ONE POSITION MUST PLAY THREE ROLES AT ONCE, SO ONE WIDE MATRIX CASTS ONE ROW INTO THREE
matrix Wc : six rows, eighteen columns (drawn fresh) -- left six -> q, mid six -> k, right six -> v
[ 0.59 0.80 -0.67 -0.49 0.29 -0.66 | -0.50 0.13 -0.59 0.51 0.64 -0.84 | 0.06 0.53 0.86 -0.41 -0.60 0.68 ]
[ 0.74 -0.54 -0.11 0.39 0.62 -0.60 | 0.30 0.55 0.09 -0.60 -0.84 -0.39 | 0.55 -0.82 -0.89 -0.25 -0.79 -0.63 ]
[ -0.86 0.04 0.35 -0.13 -0.66 -0.30 | 0.16 0.79 0.89 -0.47 -0.88 0.60 | 0.77 -0.07 0.49 0.66 0.20 0.67 ]
[ -0.86 -0.41 -0.40 -0.68 0.74 -0.85 | 0.31 -0.77 -0.25 -0.15 -0.57 0.04 | 0.06 -0.33 0.43 -0.61 -0.55 -0.26 ]
[ -0.22 -0.53 0.75 0.59 -0.71 -0.23 | -0.48 -0.09 -0.40 0.00 0.76 -0.21 | 0.27 0.17 0.45 -0.79 0.44 0.80 ]
[ 0.19 -0.38 0.31 0.38 0.28 -0.64 | 0.85 0.82 -0.14 0.17 -0.83 0.88 | 0.57 0.25 0.47 -0.56 -0.35 -0.46 ]
feed a normalized row a = [ 0.17 -0.73 0.71 -0.07 -0.10 -0.71 ]
q = a Wc[left] = [ -1.10 0.91 -0.05 -0.74 -1.05 0.65 ]
k = a Wc[mid] = [ -0.77 -0.34 0.62 0.08 0.65 -0.04 ]
v = a Wc[right] = [ -0.28 0.47 0.74 1.10 0.86 1.32 ]
one entry by hand: q[0] = a . Wc[:,0]
= 0.17(0.59) + (-0.73)(0.74) + 0.71(-0.86) + (-0.07)(-0.86) + (-0.10)(-0.22) + (-0.71)(0.19)
= -1.10
Wall: a position must hunt for helpers, advertise to others, and hand out a payout -- three jobs
at one instant -- yet it holds one row of six. One wide matrix of eighteen columns, split three
ways, casts one row into three fresh rows: a want-row q, an offer-row k, a payout-row v. One
product yields all three.
--------
TWO ROWS MUST COLLAPSE INTO ONE AGREEMENT NUMBER, SO MULTIPLY PAIRWISE AND SUM, THEN CANCEL LENGTH
want-row q = [ 0.33 0.57 0.23 -0.46 0.51 -0.64 ]
offer-row k = [ 0.59 0.15 -0.38 0.02 0.23 -0.43 ]
pairwise product and sum:
0.33(0.59)+0.57(0.15)+0.23(-0.38)+(-0.46)(0.02)+0.51(0.23)+(-0.64)(-0.43)
= 0.19 +0.09 -0.09 -0.01 +0.12 +0.28 = 0.58
cancel length: 0.58 / root(6) = 0.58 / 2.45 = 0.24 = agreement number
Wall: a want-row and an offer-row must yield one number rating fit, yet each holds six. Multiply
coordinate against coordinate and sum: agreement climbs when signs align, falls when they
clash -- a dot product. A longer row inflates that sum by length alone, so divide by root of
width; a fit rating must not ride on width. Name: scaled dot-product score.
--------
AGREEMENT NUMBERS RUN ANY SIGN, YET A BLEND NEEDS POSITIVE WEIGHTS TOTALLING ONE, SO EXPONENTIATE THEN DIVIDE BY SUM
four fresh agreement numbers:
scores = [ 0.62 -0.14 0.71 0.60 ]
exp = [ 1.86 0.87 2.03 1.82 ] (exp kills sign, keeps order)
sum(exp) = 1.86 + 0.87 + 2.03 + 1.82 = 6.58
weights = exp / 6.58 = [ 0.28 0.13 0.31 0.28 ] sum = 1.00
pressing one mark, only one agreement number exists:
weight = exp(s) / exp(s) = 1.0 (a quantity over itself)
Wall: agreement numbers run negative and unbounded, yet a blend needs weights that stay positive
and total 1 -- raw numbers fail both. Raise e to each (positive always, order kept), divide by a
running sum (totals 1). Name: softmax. Feeding one mark, one offer-row exists, so a lone weight
equals a quantity over itself, 1 -- always, at every room. No blending choice arrives at length
one; a lone mark pulls in only a payout it built.
--------
A WEIGHT OF ONE KEEPS A PAYOUT-ROW WHOLE, YET A SQUARE MATRIX MUST RESHAPE IT BACK INTO A SIX-WIDE STREAM
matrix Wo : six rows, six columns (drawn fresh)
[ -0.72 0.26 -0.34 0.46 0.08 -0.08 ]
[ 0.71 -0.80 0.10 -0.31 -0.84 0.46 ]
[ 0.11 0.71 0.18 -0.29 0.87 -0.69 ]
[ -0.81 0.42 -0.23 -0.25 0.68 -0.31 ]
[ 0.70 0.26 -0.31 -0.79 -0.46 0.84 ]
[ -0.17 -0.61 -0.36 0.72 -0.60 0.50 ]
blend (weight 1 times payout-row) v = [ -0.66 0.83 0.05 -0.82 0.78 -0.26 ]
y = v Wo = [ 2.32 -0.78 0.36 -1.17 -1.47 1.18 ]
Wall: a weight of 1 leaves a blend equal to a lone mark's payout-row -- no cross-mark mixing, for
no other mark exists. Yet a round does not halt. That blend runs through a square matrix Wo,
turning [ -0.66 0.83 0.05 -0.82 0.78 -0.26 ] into [ 2.32 -0.78 0.36 -1.17 -1.47 1.18 ]. So a lone
mark, with nobody to blend, is reshaped anyway -- by Wo now, an add next, a private reshape soon.
No blending at length one, far from idle. Name: output projection.
--------
TWELVE ROUNDS OF REWRITING COULD ERASE OR EXPLODE A ROW, SO ADD ONTO A STREAM, NEVER REPLACE
x = [ 0.42 0.04 -0.73 -0.71 -0.63 -0.61 ]
y = [ -0.81 -0.82 0.81 -0.74 0.02 -0.69 ]
-------------------------------------------------
x+y = [ -0.39 -0.78 0.08 -1.45 -0.61 -1.30 ]
Wall: twelve rounds that overwrite a row could rub out what mark 7 carried, or let numbers
balloon past control. Add a round's output onto a running row rather than replace it; a clear
road stays open for early numbers to survive twelve rounds, and each round nudges instead of
erasing. Name: residual stream.
--------
BLENDING ACROSS POSITIONS NEVER LETS A LONE ROW REVISE ITSELF, SO WIDEN, BEND EACH, NARROW BACK
matrix Wf : six rows, twenty-four columns (drawn fresh)
[ -0.51 0.47 0.79 -0.05 -0.89 0.90 -0.81 -0.20 0.07 0.71 0.58 0.20 -0.18 0.60 0.66 0.67 0.39 -0.72 -0.36 -0.03 0.01 0.60 -0.47 0.30 ]
[ -0.23 0.50 -0.55 -0.04 -0.71 -0.52 0.78 -0.33 0.71 0.04 -0.83 0.53 -0.81 0.59 -0.88 0.31 -0.60 -0.28 0.81 -0.02 0.28 0.43 -0.70 0.61 ]
[ 0.75 -0.62 0.08 -0.39 0.43 -0.85 0.02 0.53 0.42 -0.71 0.60 -0.47 0.55 -0.01 -0.61 0.42 0.55 0.36 0.84 -0.44 -0.48 -0.63 0.07 -0.18 ]
[ -0.26 -0.07 -0.41 -0.89 -0.05 -0.40 -0.09 0.77 -0.58 0.25 0.26 -0.39 -0.05 0.85 -0.28 0.38 0.61 -0.25 0.88 0.23 0.00 0.39 -0.14 0.38 ]
[ -0.41 0.77 0.08 -0.47 -0.11 0.71 0.55 -0.03 -0.27 -0.22 0.87 -0.70 0.70 0.64 -0.50 -0.27 0.13 0.54 0.81 0.89 -0.39 -0.55 -0.02 0.10 ]
[ 0.28 -0.71 0.26 0.76 0.75 0.80 0.76 0.83 -0.81 0.75 -0.41 -0.32 -0.07 0.82 0.19 -0.27 -0.23 0.73 0.58 -0.65 -0.45 -0.23 -0.51 0.11 ]
b = [ -0.46 0.67 0.36 0.64 -0.09 0.03 ] (a fresh normalized row)
wide = b Wf (twenty-four numbers):
[ 0.23 -0.24 -0.96 -0.65 0.09 -1.36 0.82 0.58 0.22 -0.35 -0.53 -0.10 -0.36 0.63 -1.24 0.31 -0.01 0.09 1.52 -0.11 0.03 0.08 -0.33 0.44 ]
bend each (large positives pass near whole, negatives fold toward 0, every number moves):
[ 0.14 -0.10 -0.16 -0.17 0.05 -0.12 0.65 0.42 0.13 -0.13 -0.16 -0.05 -0.13 0.46 -0.13 0.19 -0.01 0.05 1.42 -0.05 0.02 0.04 -0.12 0.30 ]
matrix Pm : twenty-four rows, six columns (drawn fresh) -- first four rows shown, all twenty-four multiply
[ -0.50 0.13 0.78 0.63 -0.72 -0.53 ]
[ -0.16 0.26 -0.20 -0.45 -0.28 -0.53 ]
[ 0.65 0.36 -0.16 0.36 0.04 0.05 ]
[ -0.24 -0.02 0.87 0.51 0.59 -0.68 ]
... twenty more rows ...
m = bend Pm = [ -1.52 -1.63 -0.32 0.07 0.74 0.09 ]
Wall: blending only mixes across positions, and at length one there are no others, so a lone row
can never reconsider itself by itself. Two matrices back to back fold into one matrix, buying
nothing. Widen six into twenty-four for elbow room, bend each number so two matrices cannot fold
into one, narrow twenty-four back to six to re-enter a stream. Bend name: GELU. Block name:
feed-forward.
--------
ONE ROUND BARELY MOVES A GUESS, SO STACK TWELVE ROOMS, EACH FRESH, AND WATCH A STREAM DRIFT
in [ -0.76 0.50 -0.11 0.40 0.86 0.07 ]
r 1 [ -1.88 -0.50 0.76 3.87 7.41 -0.71 ]
r 2 [ -3.22 0.80 3.39 5.43 5.51 1.15 ]
r 3 [ -1.91 3.26 4.87 6.55 0.46 -3.93 ]
r 4 [ -4.83 -1.72 4.59 6.89 -0.92 -4.90 ]
r 5 [ 0.46 -4.82 4.54 9.51 -2.06 -3.50 ]
r 6 [ -0.89 -6.77 5.21 7.60 -1.31 -7.34 ]
r 7 [ -0.00 -5.60 6.25 6.63 1.34 -8.23 ]
r 8 [ 8.99 -11.20 5.13 4.60 -0.45 -7.75 ]
r 9 [ 12.17 -13.13 -0.28 3.06 -2.03 -6.18 ]
r10 [ 12.99 -16.70 -0.64 2.16 -1.90 -7.22 ]
r11 [ 10.67 -14.05 -1.59 1.58 -2.81 -11.44 ]
r12 [ 12.14 -13.84 0.01 7.92 -5.35 -10.18 ]
Wall: one round nudges a six-number row a little, far short of a confident guess. Stack twelve
rooms, each holding fresh matrices, each a flatten-blend-project-add then flatten-widen-bend-
narrow-add. A first coordinate climbs -0.76 -> 12.14, a second falls 0.50 -> -13.84. All that
drift came from output projections and private reshapes; at length one no room ever blended
across positions. Heavy work, zero blending -- a point missed often. (These twelve rows, and
every number below, print from a program shown at close of page.)
--------
A SIX-WIDE ROW IS NOT A RANKING OVER MARKS, SO A MATRIX SCORES EVERY MARK, THEN BIGGEST WINS
flatten a last time: norm(x) = [ 1.47 -1.32 0.17 1.02 -0.41 -0.93 ]
matrix Wlm : six rows, ten columns (drawn fresh) -- one column scores one mark
[ 0.36 0.56 -0.13 -0.38 -0.66 0.77 -0.33 0.31 -0.03 0.62 ]
[ -0.31 0.62 0.74 0.61 -0.31 -0.36 -0.13 0.51 0.27 -0.44 ]
[ -0.06 -0.47 0.56 0.66 -0.28 0.82 0.02 -0.19 -0.83 -0.30 ]
[ 0.64 0.39 0.77 -0.84 0.52 0.06 0.80 0.46 0.03 0.28 ]
[ 0.83 -0.74 -0.53 -0.16 -0.35 0.10 0.18 0.32 -0.88 0.66 ]
[ -0.56 -0.74 0.56 -0.50 -0.41 -0.33 -0.49 0.33 -0.58 0.67 ]
scores = norm(x) Wlm:
mark: 0 1 2 3 4 5 6 7 8 9
[ 1.76 1.31 -0.59 -1.58 0.44 2.07 0.88 -0.22 0.39 0.84 ]
^ 2.07 biggest, at mark 5
biggest wins -> next mark = 5
Wall: six numbers are not a ranking over ten marks. A matrix of six rows and ten columns, one
column per mark, turns six numbers into one score per mark. Biggest-wins picks mark 0 here -- a
greedy rule. Often a guesser instead softmaxes scores into weights and draws by lot, so a lower
mark can surface; wording drifts run to run for that reason. Name of a score matrix: language-
model head. And a shipped machine of a hundred-twenty-three-million numbers, fed a lone "h",
scores a full stop highest -- betting a sentence is already closed. A fact about one trained
machine, not a law.
--------
PROOF -- RUN IT, GET THESE NUMBERS
Every matrix above printed from a running program with fresh draws; a reader reproduces any room
by feeding a six-number row through a flatten, a wide matrix cut three ways, a dot over root-six,
a softmax (weight 1 at length one), an output matrix, an add, a flatten, a widen, a bend, a
narrow, an add. A twelve-room run:
```python
import numpy as np, math
np.random.seed(7)
R = lambda *s: np.round(np.random.uniform(-0.9, 0.9, s), 2)
gelu = lambda v: np.array([0.5*z*(1+math.erf(z/math.sqrt(2))) for z in np.ravel(v)])
flat = lambda v: (v - v.mean()) / math.sqrt(((v - v.mean())**2).mean() + 1e-5)
D, H = 6, 24
x = R(D)
print("in ", [round(float(z),2) for z in x])
for i in range(12):
Wc, Wo, Wf, Pm = R(D,3*D), R(D,D), R(D,H), R(H,D) # fresh room
a = flat(x); q,k,v = (a@Wc)[:D], (a@Wc)[D:2*D], (a@Wc)[2*D:]
w = math.exp(float(q@k)/math.sqrt(D)) / math.exp(float(q@k)/math.sqrt(D)) # =1 at length one
x = x + (w*v) @ Wo # blend, project, add
x = x + gelu(flat(x)@Wf) @ Pm # widen, bend, narrow, add
print(f"r{i+1:>2}", [round(float(z),2) for z in x])
Wlm = R(D,10)
print("scores ", [round(float(z),2) for z in flat(x)@Wlm])
print("next mark =", int(np.argmax(flat(x)@Wlm)))
```
Running a program prints a stream drifting room by room, then a mark. Swap these small random
matrices for a hundred-twenty-three-million learned ones over a near-fifty-thousand pairing, and
an identical path lands a lone "h" on a full stop.
--------
A LADDER RUNG -- WHAT FORCED THIS PAGE, AND WHAT IT FORCES NEXT
A rung below built a smallest machine that writes at all -- one room, three turns, a line
7 8 1 2 worked by pencil: A Writing Machine of One Room. One
room barely bent a strip, so this page stacked twelve and watched drift. A wall remains:
every turn rebuilds every old row from a blank start, and a rebuilt row matches digit for
digit -- a note that kills that waste is
A Note a Machine Keeps So It Stops Redoing Old Work. A whole
ladder, rung by rung, sits at Build a GPT, Forced.