==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
Plain notes on how machine-learning rules actually work. Each one is stripped of
jargon, drawn first as a picture, solved by pencil, then written in code -- with the
textbook labels stuck on only at the very end. No hype, no frameworks, no JavaScript.
This is not a pile of loose articles -- it is a short BOOK in twenty-three chapters (sixty-nine posts),
meant to be read top to bottom. Each post opens with a "post N" path header and a
next-arrow, so you can read the whole thing front to back without ever hunting for what
comes next. Each chapter draws its ideas by hand first; the runnable Python is gathered
at the end of every post, and assumes almost no Python. Appendices tie it together.
>> NEW HERE? Start at post 1 of 69:
Guessing House Prices, End to End
No prior knowledge required. Just follow the pencil, and follow the next-arrows.
>> THE 1950 CONTRACT: print these pages and work them with a pencil -- nothing in the
teaching ever needs a computer. Where arithmetic gets heavy, imagine a room of
tireless clerks; every cost is counted in clerk-steps, and every number is
recomputed where it is needed, never remembered. Full contract on the
about page.
>> SPEAK FLUENT BUZZWORD? The Glossary / Decoder Ring translates every
plain term on this blog into its textbook name -- and back again.
CHAPTER 1 . PREDICTING HOUSE PRICES -- supervised regression from scratch
-------------------------------------------------------------------------
Part 1 -- Guessing House Prices, End to End
post 1/31 . 2026-06-01 . part 1 of 3 . fundamentals, supervised-learning
The whole picture on the California housing pile: why 20% of rows are locked away
first, the ask-closest rule and the straight-stick rule introduced plainly, RMSE
against the always-average baseline, and picking settings by rotating folds.
Part 2 -- Ask the Closest Rows: Gap, Same-Ruler, and How Slow It Gets
post 2/31 . 2026-06-01 . part 2 of 3 . knn, complexity, scaling
The ask-closest rule from the gap formula up: why a column measured in thousands
runs the whole show and must be put on the same ruler, the cost per guess and why
close stops meaning close as columns pile up, k as the stiff-vs-jumpy dial.
Part 3 -- The Straight-Stick Rule: Dials, Leftovers, and Numerics
post 3/31 . 2026-06-01 . part 3 of 3 . linear-regression, least-squares, numerics
From the bowl-shaped squared leftover to the exact dials: a three-row pencil solve
of the flat-point equations, right-angle leftovers and the flat-shadow thrower, the
Gauss-Markov conditions, and why you cut the sheet (QR/SVD) rather than flip it.
CHAPTER 2 . GRADING A GUESSER -- how good is the guess?
------------------------------------------------------
Part 1 -- Two Rulers for One Guess: MSE and R^2
post 4/31 . 2026-06-04 . part 1 of 2 . evaluation, mse, r-squared
The same straight-stick rule on a sheet of cars, graded two ways: MSE (RMSE with the
root left off) and R^2 -- a race against a fool who only ever shouts the average.
Why you divide by the whole wobble, why test R^2 can dip below zero, and the
stiff-vs-jumpy split of the miss-size.
Part 2 -- Reading the Dials: What the Coefficients Say
post 5/31 . 2026-06-04 . part 2 of 2 . coefficients, interpretation, scaling
Each column keeps one dial -- plus pushes up, minus drags down. Zip the names back
on, find the most-negative (a MIN, not a MAX), and watch the trap: raw dials wear
each column's own units, so they are not comparable until the columns sit on one
shared ruler.
CHAPTER 3 . SORTING INTO BINS -- yes-or-no guessing, scored every way
--------------------------------------------------------------------
Part 1 -- The S-Curve, the Four-Box Table, and Why Accuracy Lies
post 6/31 . 2026-06-05 . part 1 of 4 . logistic-regression, classification, evaluation
The shift from "guess a number" to "guess a bin": the S-curve squash, the
cross-entropy leftover, and how any dial sum becomes a chance. Then the four-box
table of CAUGHT/ALARM/MISSED/CLEAR, four scores that read different corners of it,
and why a machine that catches nobody can still score 63% accuracy.
Part 2 -- The Trade Curve: Sliding the Cutoff and What AUC Measures
post 7/31 . 2026-06-05 . part 2 of 4 . roc-curve, auc, cutoff
One cutoff judges the machine at one spot; slide it through every position and you
get the trade curve (ROC). The area beneath it -- AUC -- compresses the whole curve
to one number: pick a random sick lump and a random well lump; AUC is the chance the
machine scored the sick one higher.
Part 3 -- Leash and Cloud: L2 Punishment and the Two-Cloud Wall
post 8/31 . 2026-06-05 . part 3 of 4 . l2-regularization, lda
Two ways to humble an over-confident machine. L2: add a squared-dial price to the
leftover so no single dial dominates (C = 1/lambda -- small C is a heavy squeeze).
LDA: forget rolling downhill; read the two cloud centres and their shared spread,
then solve for the wall in one closed-form step.
Part 4 -- Picking Settings, Skewed Piles, and Averaging Many Classes
post 9/31 . 2026-06-05 . part 4 of 4 . grid-search, precision-recall, class-imbalance
Grid-hunting settings by rotating folds; why pinch-to-fit scaling breaks around
outliers; the precision-vs-recall trade in fraud detection; precision-recall curves
for when one class is 100x rarer; and macro vs micro averaging for 3+ bins.
CHAPTER 4 . HUMBLE DIALS AND WOBBLE BANDS -- regularisation, and how much to trust a dial
----------------------------------------------------------------------------------------
Part 1 -- The Leash: Ridge, Lasso, and Humbling the Dials
post 10/31 . 2026-06-07 . part 1 of 3 . ridge, lasso, regularisation
Free dials grow huge and memorise; put them on a leash. Ridge (the square fine) shrinks
all dials and zeroes none, solving in one step. Lasso (the absolute fine) snaps weak
dials to exactly zero and hands you a shortlist of columns. Pick the knob by the valley.
Part 2 -- One Dial Is a Lie: Bootstrap, Wobble Bands, and the Free Exam
post 11/31 . 2026-06-07 . part 2 of 3 . bootstrap, confidence-interval, out-of-bag
A single dial value hides how shaky it is. Fake 200 hospital visits by re-dealing the
pile with repeats, and read off a 95% wobble band per dial (chop 2.5% each end). The
~37% left out of each deal become a free, honest exam.
Part 3 -- The Dial by Hand: Where the Dials Really Come From
post 12/31 . 2026-06-07 . part 3 of 3 . least-squares, collinearity, derivation
The whole machine on a blank sheet: the dials are (X^T X)^-1 X^T y, the untangler that
shares credit between overlapping columns. The one-column slope is a LADDER that matches
only when columns don't overlap -- name it out loud before you build anything on it.
CHAPTER 5 . QUESTION CHARTS AND COMMITTEES -- trees, pruning, and ensembles
--------------------------------------------------------------------------
Part 1 -- Question Charts: Building a Tree by Hand
post 13/31 . 2026-06-07 . part 1 of 3 . decision-tree, regression, classification
No dials, no formula: a machine that asks yes/no questions. Built from scratch --
sort each column, try every midpoint, pick the cleanest cut, recurse. Five wrong
pictures corrected: one column per split, depth is people not columns, build and
use are separate phases, same column may recur at a tighter cut.
Part 2 -- The Mixing Ruler: Gini, Information Gain, and Pruning
post 14/31 . 2026-06-07 . part 2 of 3 . gini, information-gain, pruning
Flip to the cancer sick/well sheet: the badness ruler swaps from squared misses to
Gini (chance two random grabs disagree) -- derived by hand from four counts. Then
pruning: a tax per leaf snips weak branches after the tree is fully grown; the alpha
menu comes from the tree itself; the nested-averages ladder picks the best tax.
Part 3 -- Committees: Bagging, Random Forest, and Boosting
post 15/31 . 2026-06-07 . part 3 of 3 . bagging, random-forest, boosting, ensembles
One chart memorises and panics; 200 averaged charts are eerily steady. Bagging
re-deals with repeats and averages 200 trees (OOB free exam built in). Random forest
hides a handful of columns at each cut so the trees truly differ. Boosting chains
stumps in a line, each fixing the last one's leftovers. Two ways to interrogate the
black box: scramble a column (permutation importance) and slide a column (PDP/ICE).
APPENDICES . FLIP-TO REFERENCES -- every term in one place
----------------------------------------------------------
A. Classification Reference -- Loss, Leash, Grid, and All the Terms
reference . 2026-06-06 . companion to Chapter 3 . log-loss, lda, grid-search, pr-curves
All classification concepts from Chapter 3 in one flip-to reference: cross-entropy vs
MSE, the C parameter, LDA with priors, hyperparameters vs dials, GridSearchCV fold
safety, min-max scaling, precision vs recall in business, ROC vs PR curves, skewed
piles, and macro/micro/weighted averaging -- plain language first, standard labels
last. Read Chapter 3 first; use this as a reference when you need to look something
up quickly.
B. Distance and Clustering Reference -- Rulers, Traps, and Ethics
reference . 2026-06-09 . companion to Chapter 6 . hamming, mahalanobis, missing-data, ethics
The loose ends Chapter 6 does not stop for: the k you pick, two more rulers (Hamming for
words, Mahalanobis for stretched many-walled rooms), the missing-data traps, why crush a
room at all, and the part the lab skips -- the ethics of sorting PEOPLE into piles (bias,
privacy, transparency) and what segmentation is really for.
C. LSTM From Pencil -- RNN and LSTM From Scratch, Nothing But a Pencil
reference . 2026-06-14 . companion to Chapter 10 . lstm, rnn, by-hand, kata
A KATA: one clean, self-contained walk through the RNN and the LSTM by pencil, made to be
redone on a blank sheet again and again until your hand knows it. Words into 32-number
rows, grid-times-row by hand, the RNN's one-memory recipe and why it FADES (a theorem),
then the two-memory fix -- a silent uncrushed vault A and a spoken readout B, four machines
(FRESH + keep/admit/show), the combine worked in real numbers (5.0 -> 4.62 -> 0.70) -- plus
a struggle log of every trap and a one-word end-to-end recitation. No computer, no code.
D. Transformer From Pencil -- Attention From Scratch, One Number at a Time
reference . 2026-06-15 . companion to Chapter 10 . transformer, attention, by-hand, kata
A KATA: one clean, self-contained walk through the attention mechanism by pencil, made to
be repeated on a blank sheet until your hand knows it. Words into a dictionary, padded rows,
sticks from an embedding table; each word makes WANT/HAVE/GIVE from three reused grids;
nolan's WANT dots every HAVE → scaled by √(tag width) → softmax → portions; portions weight
GIVE sticks → ADD slot by slot → nolan_new = [0.094, 2.859, 0.953, 0.047] fully worked.
Multi-head flagged, positional-encoding gap flagged, four WRONG TURN boxes, one-breath
recitation at the end. No code; pencil and real numbers only.
E. Vision Transformer From Pencil -- Strips, Seat-Stamps, and Masks by Hand
reference . 2026-06-15 . companion to Chapter 11 . vit, layernorm, masking, kata
A KATA: the pencil-able cores of the vision transformer, redone on a blank sheet until your
hand knows them. A tiny 4x4 photo cut into strips (fold/swap/glue, and the all-black bug a
missing swap causes), the sine/cosine seat-stamp worked at three seats, the attention grid
on two strips, LayerNorm on one row (45, √125, [0.45, 1.34, -0.45, -1.34]), and the causal
mask triangle of -inf that splits reader from writer. Drills throughout. No code.
SPECIAL. Genetic Algorithm From Scratch -- Optimising With No Gradient, Every Number Shown
special . 2026-06-18 . standalone . genetic-algorithm, optimisation, evolution, by-hand
Written for Hacker News, self-contained: a genetic algorithm built from nothing on a toy
hunt -- secret [1,3,2,1], a population of 4 guesses -- with every number worked. The score
as one blind "how wrong" number (total gap, negated); the flip-and-share trap (dividing raw
errors breeds the worst, 1-x goes negative, so subtract the worst score then divide to sum 1
-> chances 0.375/0.25/0.375/0); weighted-with-replacement wheel selection by cumulative
slices; one-point crossover (cut g1xg3 at k=3 -> [1,3,2,1], the target) shown to be BLIND;
single-slot mutation on a coin; babies replacing the whole population. The framing insight:
it optimises with NO gradient, only a ranking plus letting losers die. Honest notes on the
gradient-free tradeoff, legal flip-shapes (1/(1+sin), e^-sin), and why not just the top two.
Working numpy code. Assumes zero background; every term defined where it appears.
SPECIAL. Cheapest Walk: UCS and A* -- Every Pop Shown, the Proof Included
special . 2026-06-17 . standalone . search, ucs, a-star, dijkstra, by-hand
Written for Hacker News, self-contained: the full UCS walk (6 pops) on a 5-node town map
(S, A, B, C, G), every offer and discard shown with arithmetic; then the same map in A*
with positions assigned to nodes and every f=g+h worked out, demonstrating the exact pop
A* saves over UCS (stale B never surfaces because G's f=5.00 beats B's f=6.24). Proof that
the cheapest pop is final (any alternative costs ≥ c + 0 = c), the admissibility argument
(h ≤ true remaining cost, always), honest note that UCS IS Dijkstra's, lazy deletion
explained, and working heapq code for both. Assumes zero background; every term defined
where it appears.
SPECIAL. Informed Search Jargon -- One Machine, Three Slips, Every Word Debunked
special . 2026-06-18 . standalone . search, a-star, heuristics, ucs, greedy, by-hand
Written for Hacker News, self-contained: the jargon companion to the UCS/A* post. One
machine (a cheapest-pile), three slips (g for UCS, h for greedy, g+h for A*), and the
exact reason each differs from the others -- walked on a 4-node toy town (S, A, B, G;
two paths: S->A->G=101 and S->B->G=20). Greedy picks S->A->G=101 (fooled because h(A)=1
"looks" close while the 100-toll road behind it goes unnoticed); A* picks S->B->G=20
(g corrects the lie the moment A* walks the toll). Admissibility debunked: the guess never
lies HIGH (underestimating is safe; overestimating is the only flaw). Consistency as the
triangle rule worked by hand (h(n) <= c + h(n')). Three ways a search blows optimality
(never finishes / not chasing cheapest / map shifts). A*'s fat-ring memory vs DFS's thin
stick. Good heuristic traits vs the fakes. Full decoder sheet. Working Python with
hard-coded round-by-round variables.
SPECIAL. Bayes by Head-Count -- The Greedy Shopkeeper, His Faulty Machine, and the Sick in the Wild
special . 2026-06-19 . standalone . probability, bayes, statistics, by-hand
Written for Hacker News, self-contained: Bayes' theorem built from one hospital corridor
rather than a formula. A patient feels unwell; the doctor writes a referral and sends them
down the corridor to the greedy shopkeeper, who runs a faulty test machine -- each BEEP is a
sale, so he hides the total count (that hidden total is P(Beep)). The machine lies two ways
(a miss = false negative -> the patient walks home into the wild undiagnosed; a glitch =
false positive -> a healthy patient comes back alarmed). The doctor asks "given this BEEP
paper, how sick is this patient?" The one move: turn every fraction into PATIENTS IN THIS
GROUP; then every Bayes question is "of the patients who came back with a BEEP paper, how many
are truly sick?" Ten VISITS worked by head-count: joint vs conditional (comma = whole group,
bar = back-room only), Law of Total Probability as gluing two beep-piles, the rarity trap (1%
disease + 10% glitch = 8.3% precision despite a 90% catch rate), a two-cause machine (Disease
+ harmless Sniffle floods the pile), two machines in series (precision 76% but more sick
missed), a useless red-coat distractor the doctor must learn to ignore. The honest split:
Bayes answers the doctor's question (given a beep, how sick?), NOT the public-health question
(how many did we miss?). Python: visits hard-coded line by line + a full simulation whose
variable names match the story. Full decoder sheet (prior/likelihood/posterior/sensitivity/PPV).
Assumes zero background; every term defined where it appears.
SPECIAL. The Walking Machine and the Vault -- RNN and LSTM by Pencil
special . 2026-06-20 . standalone . rnn, lstm, sequence-models, by-hand
Written for Hacker News, self-contained: why the plain RNN breaks and why the LSTM fixes it
-- with real arithmetic at every step, not a diagram. The RNN walks "nolan ended" word by
word, crushing one memory through tanh on every carry; word 1's signal is 0.664, word 2 it
is 0.581, by word 8 it is 0.377 and falling -- "not good" becomes "good" because "not" is
gone. The LSTM fix: TWO memories -- a silent uncrushed vault (A, free to hold 5.0 across
words) and a spoken readout (B). Four machines (FRESH + keep/admit/show) each read word +
old B; the combine new A = keep*old_A + admit*FRESH grows the vault without an unconditional
crush; new B = show*tanh(new A) tames the vault for the outside world. Worked with one number:
old A=5.0, keep=0.71, admit=0.41 -> new A=3.62, still large. Case 3 runs both over 10 words:
RNN memory is below 1.0 by word 1 and stuck there; LSTM vault is 1.74 after 10 words from 5.0.
Honest footnote: width-1 example generalises element-wise to the real 32; keep is learned, not
fixed. Full Python, hardcoded, output verified. Assumes zero background.
SPECIAL. Attention and the Transformer by Pencil -- Every Word Looks at Every Word
special . 2026-06-20 . standalone . transformer, attention, self-attention, by-hand
Written for Hacker News, self-contained: the Transformer built from scratch by following ONE
word ("nolan") through every move with real numbers. Why walking fades (RNN: 79 rewrites to
tie word 1 to word 80) vs look-across (direct, zero rewrites). Why THREE sticks per word:
WANT to look, HAVE to be looked-at, GIVE to hand over -- one embedding cannot serve all three
roles. How a stick is made from a grid (matrix-vector, derived). Match: nolan.WANT=[2,0,1,0]
dotted against every word's HAVE -> nolan->nolan=2, nolan->ended=8. Scale: divide by sqrt(4)=2
-> [1,4]. Softmax: e^1=2.718, e^4=54.60, total=57.32, fractions=[0.047, 0.953] (adds to 1;
exponential sharpens the winner). Weighted sum: 0.047*[2,0,0,1]+0.953*[0,3,1,0]=[0.094,2.859,
0.953,0.047] -- nolan is now mostly "ended." Two teams (multi-head): two want/have/give sets,
results glued. Position: no walking = no order -- positional encoding is what fills the gap
(flagged as skipped in the lab). Collapse 100 sticks -> one by averaging. Plain verdict head:
Dropout+Dense(20,relu)+Dropout+Dense(1,sigmoid). Python: the full numpy attention trace (every
number matches the pencil), then the 8-line Keras transformer model, each line mapped to its
pencil move, plus binary cross-entropy derived (truth=1, said 0.96 -> -log(0.96)=0.04). Honest
footnote: encoder-only (no future masking, no cross-attention); positional encoding named and
flagged as skipped. Assumes zero background.
SPECIAL. The Mark That Tells a Transformer Where It Is -- Positional Encoding Built From Scratch
special . 2026-06-22 . standalone . transformer, positional-encoding, rope, by-hand
The companion to the attention post that fills the gap it flagged as "skipped." Two naive
marks (scalar, normalised) fail for specific reasons worked by arithmetic. Sinusoidal encoding
built from scratch: exponential frequency spacing, why both sin and cos are needed, every
number computed for positions 3 and 4. Three sinusoidal failure modes. Then RoPE: the rotation
formula derived, "cat" at positions 3 and 7 rotated by hand, and the full algebraic proof that
q_m · k_n = (q·k)·cos((m−n)θ) + (q×k)·sin((m−n)θ) -- absolute positions cancel, only the
gap remains, forced by geometry not learned. Slow-pair failure shown with angles (47° training
range vs 298° at inference). Base-frequency scaling fix computed (0.000104 → 0.0000021, ~49x
slower, 6° vs 298° for gap 50,000). Production recipe: geometry fix + continued training +
long-range data. Python unrolled for d_model=4, 512-dim loop, all scores verified.
SPECIAL. The Window That Drops Its Best Old Match -- Sliding Window Attention by Pencil
special . 2026-07-09 . standalone . transformer, attention, sliding-window, causal-mask, by-hand
The companion to the attention post that works the masking it skipped. Full attention costs
the square of the row length (100,000 tokens -> 5 billion matches), so cap each token to its
last W. Worked on token 5 of six, W=3: its scores against tokens 0..5 come out 9,8,7,2,4,6 --
the loudest match, 9, is the OLDEST token. Full attention hands token 0 a 64.1% share; the
window strikes tokens 0,1,2 and that same best match drops to 0.0%, thrown away only for
being old. The kept-set is two number-line tests (j <= i not-ahead AND j >= i-W+1 not-stale),
a band of 3 that slides -- token 5 keeps {3,4,5}. Why strike the SCORE to minus infinity, not
the share (e^-inf = 0 keeps the survivors summing to one; zeroing after leaves the blend
short). Then the payoff: stacked window-3 layers rebuild the reach two tokens per layer, so
five layers see ten back, not three -- local per layer, not local overall -- and the seam
where that relay still breaks. Python unrolled, all shares verified.
SPECIAL. The Two Flips That Are Not the Same Flip -- Attention's Shapes by Pencil
special . 2026-07-10 . standalone . transformer, attention, tensor-shapes, multi-head, by-hand
The plumbing companion: attention's arithmetic is easy, but WHERE the numbers sit and how they
are re-grouped is where days vanish. One book, three tokens, two lenses, followed from a wide
row to a finished row. The projection coefficients live in one shared table, not stapled to the
outputs; cutting the wide row into WANT/HAVE/GIVE is scissors, no math; view only draws dividers
(width 4 -> 2 lenses of 2), moving nothing. Then the trap: ONE word, "transpose," is two
different flips. The re-shelve (transpose(1,2)) swaps the token and lens SHELVES so each lens
owns all its tokens; the inside flip (transpose(-2,-1)) stands one lens's grid on its edge,
(T,D)->(D,T), so HAVE rows become columns -- the only reason Q @ K-transpose lines up, since the
dot eats left-rows against right-columns. A dimension is a counter; a grid appears only when you
pin all but two (that is batched matmul). Token COUNT (T) is not token WIDTH (D); after the match
the width axis is gone and the share-split runs across tokens. Merge is the re-shelve run
backward plus a glue. Full multi-head sliding-window forward, every shape printed, all verified.
SPECIAL. Sliding Window Self-Attention, Built From One Pencil And One Page
special . 2026-07-10 . standalone . transformer, attention, sliding-window, from-scratch, by-hand
Starts from a blank page and a single row of numbers, builds a whole layer with nothing borrowed:
nine forced lines, every WHY worked by hand, clear as water down to a last decimal. One shared
C-by-3C table makes Q/K/V, and its 3C outputs are outputs not coefficients (why three roles: tie
Q=K and every token stares at itself); slices make heads; a query-row dotted with a stood-up
key-column makes the T-by-T score grid; root-D tames its spread -- derived in full from
expectation and variance (E[X^2]-(E[X])^2, independence E[XY]=E[X]E[Y], one product term variance
sigma^4, sum of D adds to D*sigma^4, spread sigma^2*root-D, cancel by root-D); tril + triu carve
the causal-plus-window band; a minus-infinity strike beats zero (e^0=1 still counts, worked in
numbers); softmax on the last axis splits shares across keys; shares blend value rows;
transpose+contiguous+view glue heads back to width C. Plus O(n^2)->O(n*W) by clerk-count and reach
~= L*(W-1) through stacked layers. Every number machine-checked; written with the word "the"
banned throughout.
SPECIAL. Build a GPT, Forced -- A Reading Ladder
special . 2026-07-14 . standalone . transformer, gpt, reading-order, ladder
A hub that orders the GPT specials into one forced build: each page exists because a
machine on a page below hit a wall, worked out in numbers. Rung 1: a one-room writer that
works. Rung 2: depth, because one room barely bends a strip. Rung 3: a note (KV cache),
because rebuilding old rows piles up 1+2+3=6 where three rows exist. Rung 4: a window,
because 100,000 marks cost five billion matches. Rung 5: shapes, because days vanish in
WHERE numbers sit, not in arithmetic. Rung 6: order, because look-everywhere forgets
where. Ends with a foundations shelf -- three attention-alone posts for a reader starting
cold.
SPECIAL. A Writing Machine of One Room -- Smallest GPT That Works, by Pencil
special . 2026-07-14 . standalone . transformer, gpt, forward-pass, generation, by-hand
Rung 1 of the ladder: the smallest machine that genuinely writes. One room of machinery,
six numbers per mark, ten marks known, run for three full turns until a four-mark line
exists: 7 -> 7 8 -> 7 8 1 -> 7 8 1 2. Every move forced by a wall: an integer multiplies
into nothing, so a table hands every mark six numbers; mark 7 first and mark 7 third read
one identical row, so a slot-row is added on; one loud coordinate hijacks every sum, so
center a copy and divide by spread; a strip must hunt, offer, and hand over at once, so
three matrices cast want/label/payload rows; a line of one mark matches only itself, so a
share of exactly 1 falls out; widen-bend-shrink, then a last matrix scores all ten marks.
Turn two splits shares for real; turn three shows a newest mark leaning hardest on an
oldest one. Ends where it cracks -- one room is shallow -- and a hardcoded numpy proof
reproduces every number.
SPECIAL. What a Machine Does When You Press "h" -- Every Matrix, Forced Into Being
special . 2026-07-12 . standalone . transformer, gpt, nanogpt, forward-pass, by-hand
Presses one letter "h" and walks a GPT forward pass where each operation is forced by a mechanical
wall a round earlier hits -- pair a mark to an integer, park a row of numbers per mark, add a slot-
row, flatten (LayerNorm), cast one row into want/label/payload by a wide matrix, dot over root-width
(score), exponentiate to positive weights totalling one (softmax), an output matrix, a residual add,
a widen-bend-narrow (GELU feed-forward), twelve rooms, a score matrix over every mark, biggest wins.
At length one a lone mark's weight is a quantity over itself, 1 -- no blending -- yet output matrices
and private reshapes drift a strip past 12 and below -13. Every matrix drawn fresh from a running
program so no number repeats, full large matrices, a hardcoded numpy block reproduces a twelve-room
drift and a guess. Word "the" banned throughout.
SPECIAL. What Happens When You Press "h" in a GPT -- A Plain Walkthrough
special . 2026-07-13 . standalone . transformer, gpt, nanogpt, forward-pass, explainer
The casual, shareable version: one letter "h" through a whole GPT, front to back, in plain English
with all the math and a runnable proof. Tokenize to id 71, embed, add position, then twelve blocks
of LayerNorm -> Q/K/V from one matrix -> scaled dot-product score -> softmax (a flat 1 for a single
token) -> value -> output projection -> residual add, plus a widen/GELU/shrink MLP. The single-token
punchline worked out: attention returns the value untouched, yet the running vector still drifts from
under 1.0 to past 12 -- so the projections and MLPs do all the lifting. Ends on real GPT-2 (123M),
where a lone "h" predicts a full stop. Two short numpy/torch blocks reproduce the toy drift and the
real result.
SPECIAL. Press "h" -- Final GPT Pencil Pass
special . 2026-07-13 . standalone . transformer, gpt, forward-pass, by-hand
The final standalone pencil version: one keypress through token id, token row, position row,
LayerNorm, q/k/v, head split, scaled dot-product score, one-token softmax share, value blend,
output projection, residual add, MLP widen-bend-narrow, final logits, greedy choice, and
sampling.
Every new name is earned after the arithmetic that forces it; every key row is drawn before use;
the toy is six-wide with two heads and ten marks, while GPT-2's 768-wide, 12-head, 50,257-token
shapes sit beside it. Ends with hard-coded plain Python: no random draw, no model download, no
skipped table.
SPECIAL. A Note a Machine Keeps So It Stops Redoing Old Work -- KV Cache by Pencil
special . 2026-07-12 . standalone . transformer, kv-cache, autoregressive, inference, by-hand
A machine writing one word per turn rebuilds a label-row (word-row through a fixed matrix) for every
word every turn, since it keeps no memory between turns -- yet a label-row draws on one word alone,
so word1's row prints identical at turn 1, 2, 3. Builds pile as 1+2+3=6 where three rows exist;
writing each row once on a note collapses six to three, and 125,250 to 500 across five hundred
words. That note is a KV cache. A payload-row (a second matrix) joins it; a want-row (a third) is
cast fresh for a newest word and dropped; a note is per-room. Every matrix drawn fresh from a running
program, full six-by-six matrices, a hardcoded numpy block reproduces every number. Words "the" and
"same" banned throughout.
SPECIAL. The Agent Ladder -- Five Cats, One New Power Each, From Reflex to Learning
special . 2026-06-18 . standalone . ai-agents, planning, utility, learning, by-hand . (also: Chapter 12 intro)
Written for Hacker News, self-contained: why five agent types are a LADDER (each rung =
the rung below + exactly one new power) rather than a taxonomy. Rung 1 (twitch/simple
reflex): wiring table, breaks at sealed box -- mouse hides, cat wanders off (no memory).
Rung 2 (diary/model-based): adds memory, beats hidden mouse, breaks at juke -- diary
remembers old direction, cat charges wrong way. Rung 3 (sketchpad/goal-based): adds
planning, beats juke, breaks at black-and-white -- two catching plans look identical,
no way to rank safe vs deadly. Rung 4 (scorecard/utility): adds happiness scores (+100
catch, -50 trap, -1/step), beats ranking; expected utility worked (0.80*100+0.20*(-5)=79.0
vs 0.60*100+0.40*(-5)=58.0); breaks at frozen model and math exploding. Rung 5 (learning):
adds updatable dials (belief 0.70 -> 0.56 -> 0.45 -> 0.36 over three left-chases); breaks
at cold start and raw speed ceiling. Three laws. Pattern table. Full decoder. Python: five
separate blocks, hard-coded rounds, each rung succeeds then hits its wall.
SPECIAL. Transformers With Pencil -- A Whole Block Worked by Hand, One Line at a Time
special . 2026-06-15 . standalone . transformer, attention, from-scratch
Written for Hacker News, self-contained: ONE full transformer block run end to end on two
words (cat, sat), every number by pencil. Word -> row, position stamped on, three views
(ASK/OFFER/HANDOVER = Q/K/V), dot-product scores, shrink by √width, softmax to shares
[0.378, 0.622], weighted sum of HANDOVERs, add-the-original + LayerNorm (middle 0, distance
1), a widen-bend(ReLU)-narrow worker, add + tame again -- cat goes in [2,1,1,0], comes out
[1.431, 0.349, -0.541, -1.239]. Plus stacking, multi-head, and the minus-infinity future
switch that turns a reader (BERT) into a writer (GPT). Assumes zero memory; redefines every
term where it appears; nothing waved at.
CHAPTER 6 . FINDING PATTERNS WITHOUT ANSWERS -- unsupervised learning
----------------------------------------------------------------------
Part 1 -- Looking at a Sheet With No Answers: Means, Distance, and the Ruler Problem
post 16/31 . 2026-06-09 . part 1 of 6 . eda, distance, standardization, unsupervised
No answer column, no right or wrong -- just 50 states and 3 measurements. The mean
and spread of each column, the ruler problem (Assault in hundreds drowns UrbanPop
in tens), two gap rulers (straight-line Euclidean and city-block Manhattan), the
sheet of gaps, and the fix: put every column on the same ruler and measure again.
Part 2 -- The Strongest Direction: Crushing a Many-Wall Room Into a Flat Page (PCA)
post 17/31 . 2026-06-09 . part 2 of 6 . pca, dimensionality-reduction, visualization
13 chemical measurements = 13 walls. You cannot draw a 13-wall room. Shine a
flashlight, trace the longest shadow (PC1), then the next at a right angle (PC2).
A 2-column by-hand worked example, the recipe (loadings), how much each shadow
carries (PVE), choosing how many to keep (80% threshold), and blowing the shadow
back up (reconstruction error).
Part 3 -- Grouping by Nearest Centre: K-Means From a Blank Sheet
post 18/31 . 2026-06-09 . part 3 of 6 . k-means, clustering, unsupervised
The first machine that actually GROUPS: pick K, then loop two moves -- assign every dot
to its nearest centre, move each centre to its pile's mean. Tightness as the score, the
unlucky-start trap, and picking K by the elbow.
Part 4 -- The Family Tree: Hierarchical Clustering and the Dendrogram
post 19/31 . 2026-06-09 . part 4 of 6 . hierarchical-clustering, dendrogram, linkage
No K up front: start with everyone alone and marry the two closest groups over and over,
recording each wedding's height. Read the family tree, cut it wherever you like, and the
cut decides how many groups. Single, complete, average, and Ward linkage compared.
Part 5 -- Both Tools on NCI60: PCA and Clustering on Real Gene Data
post 20/31 . 2026-06-09 . part 5 of 6 . pca, clustering, nci60, case-study
All three tools on one hard sheet: 64 samples, 6,830 genes -- more columns than rows.
Crush with PCA, carve with K-means and the family tree, then unseal the cancer types and
grade the blind groups. Peeking at labels is a scorecard, never an input.
Part 6 -- Filling the Blanks: Recommender Systems and Matrix Factorisation
post 21/31 . 2026-06-09 . part 6 of 6 . recommender-systems, matrix-factorisation
A sheet that is mostly holes: users x movies, a rating only where someone watched. Split
the holey grid into two skinny full sheets of hidden tastes and flavours -- PCA's
scores-times-recipe shape again. Then the lab end to end: humble each movie column, drop
lazy zeros, and run rebuild-restore rounds until the marks freeze -- past the hand recipe
that died of its own echo.
CHAPTER 7 . BUILDING A NEURAL NETWORK FROM SCRATCH -- deep learning, forward then backward
-----------------------------------------------------------------------------------------
Part 1 -- Stacked Rooms and One Walk by Hand: How a Network Computes a Guess
post 22/31 . 2026-06-11 . part 1 of 2 . neural-network, deep-learning, relu, sigmoid
A fully self-contained start -- no prior posts needed. One room of clerks draws straight
lines; stacked rooms draw anything, but only if a non-linear bend (the zero-out rule,
ReLU) breaks the collapse between them. The forward pass walked by hand: three-cut split,
humbling the columns, every multiplication of one patient shown, the S-curve derived from
odds, and cross-entropy loss. Ends with a built network whose dials have not yet moved.
Part 2 -- Rolling Downhill by Hand: How a Network Learns
post 23/31 . 2026-06-11 . part 2 of 2 . backpropagation, gradient-descent, adam, dropout
The dials finally learn. Backpropagation worked by hand on a one-input, one-hidden,
one-output network: the chain rule link by link, the (guess - truth) shortcut derived,
every slope checked against a brute-force wiggle, then the error sent one room further
back through the dead-clerk gate. Learning rate, Adam, dropout as co-adaptation, the
overflow clip, three real debugging mistakes, and the end-to-end run (0.974 exam).
CHAPTER 8 . KEEPING A NETWORK HONEST -- the fight against memorising
-------------------------------------------------------------------
Five Machines Against Memorising: A Tax, a Coffee Break, a Fire Alarm, and a Humbler
post 24/31 . 2026-06-12 . overfitting, regularisation, dropout, batchnorm
A network that learns too well memorises the study pile's freckles and flunks new
patients. One plain machine catches the disease on a sheet of clothing photos; four cures
each fight it differently -- a tax on big dials (L2), sending clerks home (dropout), a fire
alarm that stops at the valley floor (early stopping), and a humbler between floors (batch
norm). Softmax and its ten chances derived by hand, then all five judged on one sealed
pile, with the study-minus-practice gap naming the biggest memoriser.
CHAPTER 9 . MACHINES THAT LOOK AT PICTURES -- convolution, by hand
-----------------------------------------------------------------
Part 1 -- A Magic Paper Slid Over a Photo: How a Picture Network Sees
post 25/31 . 2026-06-12 . part 1 . cnn, convolution, pooling, computer-vision
Flatten a photo and you destroy every "next to" -- so this machine keeps the picture folded
and slides a tiny 3x3 window (a magic paper of nine dials) across it, lighting up where its
shape sits. Built one worker at a time with every wrong picture corrected: the edge-finder
that scores 60 on an edge and 0 on flat grey, 27 dials across three colours, 32 inspectors
into 32 score-sheets, the shrink boss that keeps the loudest of each 2x2, the second floor
drilling through all 32, and the punchline -- 262,208 of 282,250 dials hide in one Dense
floor because the small papers are reused everywhere. (Colour photos, CIFAR-10.)
Part 2 -- The Deep Factory: Humbler, Send-Home, and the Confusion Sheet
post 26/31 . 2026-06-13 . part 2 . cnn, batch-norm, dropout, confusion-matrix
The simple factory lands ~70%. Three layers of armour push it further: a humbler steadies
every inspector's 65,536 numbers to middle 0, scatter 1 (arithmetic done by hand, full
clerk count); send-home zeroes a random 25% of sheet entries to break secret teams (one
coin per entry, off at exam time); the deep factory inserts both in the right order --
floor → humbler → boss → send-home. Then the confusion sheet: sort 10,000 test cards
into 10 true-animal piles, sort each pile by guess, and the biggest off-diagonal cell
names cat↔dog as the most-confused pair. Ends with reading the magic-paper shape (3,3,3,32)
directly from the trained floor.
CHAPTER 10 . MACHINES THAT READ WORDS -- recurrent networks, by hand
-------------------------------------------------------------------
Part 1 -- Words Into a Machine: The Notepad and the Walking Worker
post 27/31 . 2026-06-13 . part 1 of 3 . rnn, embedding, nlp, sequence-models
A photo was already numbers; a review is WORDS, and a factory can only multiply numbers.
Number the 10,000 commonest words by frequency, pad every review to 100, then a NOTEPAD
swaps each word-number for a learned 32-number note (so "boring" and "dull" drift close,
and "not" can carry a flip). One walking worker reads the 100 notes IN ORDER, reusing one
dial-set to rewrite a 32-number memory -- the RNN cell, walked by hand word by word -- and
a final S-curve clerk reads the last memory for thumbs up or down. (IMDB sentiment.)
Part 2 -- The Two-Memory Worker: How an LSTM Remembers Far-Back Words
post 28/31 . 2026-06-13 . part 2 of 3 . lstm, bilstm, sequence-models
The plain worker crushes its one memory every word, so word 1 fades to nothing by word 90.
The fix, derived by hand: carry TWO memories -- a long keep (A, never crushed, free to grow
past 1) and a spoken recent (B) -- and let three 0..1 voters decide what to keep, admit, and
speak. That is the LSTM (the old RNN cell plus three dimmer knobs), shown with a full
real-number combine. Then the BiLSTM reads the review both directions and glues the two
memories, an honest four-way comparison, and a blunt note: the fade is a theorem, the exact
wiring is an engineered choice.
Part 3 -- The Look-Across Machine: Attention and the Transformer by Pencil
post 29/31 . 2026-06-15 . part 3 of 3 . transformer, attention, sequence-models
The walk costs two things: words run in serial (a GPU bottleneck) and far-apart words
connect only through a long rewrite chain that fades. The fix: lay all 100 words out at once
and let every word look DIRECTLY at every other, near or far, same cost, all at the same time.
Built by pencil for the review "nolan ended" (width 4, every number computed): each word makes
WANT/HAVE/GIVE tags from three reused grids; dot products score cross-word matches; scaled by
√(tag width) and softmaxed into portions; portions weight the GIVE sticks, added slot by slot
→ a richer new stick per word. Two heads catch two kinds of link. Average all sticks →
one summary → a small plain-worker head → the tick. Full code (Q8-Q10), numpy verification,
four-way comparison (SimpleRNN / LSTM / BiLSTM / Transformer), and one honest flag: no walk
means no order -- positional encoding is what the real machine adds.
CHAPTER 11 . ATTENTION GROWS EYES -- vision transformers, and the encoder/decoder split
--------------------------------------------------------------------------------------
Part 1 -- The Vision Transformer: A Photo Cut Into Strips That Look at Each Other
post 30/31 . 2026-06-15 . part 1 of 2 . vision-transformer, attention, layernorm, cnn
Attention, last seen reading words, now points at pictures -- and pays the debt Chapter 10
flagged. A 28x28 photo is cut into 49 strips of 16 dots (fold/swap/glue, and why a missing
swap prints all-black patches); each strip re-described 16->64, then stamped with WHERE it
sat -- positional encoding built by hand TWO ways, a fixed sine/cosine wave and a learned
seat-table. Every strip looks at every strip (Chapter 10's attention, the full grid worked on
two strips, four heads). Two new machines join the toolbox: the keep-the-original wire (skip
connection) and a per-strip humbler (LayerNorm, [50,60,40,30] -> [0.45,1.34,-0.45,-1.34],
contrasted with Chapter 8's BatchNorm). Then GlobalAveragePooling to one summary, a softmax
head over 10 clothing kinds, the full code, and a ViT-vs-CNN race -- with the honest reason
the CNN often wins on small piles: it comes pre-wired with "nearby goes together". (Fashion-MNIST.)
Part 2 -- Encoder or Decoder: The One Mask That Splits BERT From GPT
post 31/31 . 2026-06-15 . part 2 of 2 . bert, gpt, encoder, decoder, masking
One switch splits the whole transformer world. Lay attention scores in a grid (row looks,
column looked-at); leave every cell open and a token sees both ways -- bidirectional, a
reader that judges (the ENCODER: BERT for words, the ViT you just built for pictures). Set
every future cell to minus infinity before softmax (e^-inf = 0, so zero portion) and a token
sees only the past -- causal, a writer that generates one token at a time (the DECODER: GPT).
The mask worked by hand on three tokens, the family tree drawn, and the honest placement of
your own lab: (x,x) with no mask + a softmax-over-10 head = a bidirectional encoder for
understanding, the BERT shape pointed at images. You built the reader, never the writer.
Plus a blunt note: the original 2017 transformer used BOTH halves at once for translation.
CHAPTER 12 . REINFORCEMENT LEARNING -- agents, rewards, and plans by hand
--------------------------------------------------------------------------
Intro -- The Agent Ladder: Five Cat-Bots, One New Power Each
intro . see also: SPECIAL below . 2026-06-18 . ai-agents, planning, utility, learning
Why five agent types are a ladder (each rung = the rung below + exactly one new power).
Required reading before the RL posts. Also stands alone as a Hacker News piece.
Part 1 -- Bandits and Exploration: The Greedy Shopkeeper and His Faulty Machines
post 32/36 . 2026-06-21 . part 1 of 5 . reinforcement-learning, bandits, exploration, epsilon-greedy, by-hand
A greedy shopkeeper at a railway station clinic rents faulty machines from a factory. Three
machines, 1000 patients, and one problem: the true average payout of each machine is hidden.
Three patients traced by pencil, the running average proved without storing old payouts, the
tie-breaking bug, the greed lock (first positive estimate captures all future patients), the
100-machine version of the same trap, and the epsilon fix -- 10% random breaks the lock.
Then shrinking nudge vs fixed nudge with the overnight factory-swap test: alpha=0.1 needs
~30 patients to unlearn a stale estimate; 1/n needs hundreds. Python for all three shopkeepers.
Part 2 -- Worth and Bellman from Zero: What a Spot Is Worth When the Future Branches
post 33/36 . 2026-06-22 . part 2 of 5 . reinforcement-learning, bellman, mdp, value-function, discount, by-hand
Return as a pile of rewards, averaging unsure rewards (value × chance), the fade γ that keeps
an endless pile finite (geometric series 1/(1-γ) derived), episodic vs continuing tasks, the
world's die p(s',r|s,a), deterministic vs stochastic policies, V and Q as two separate worth-
machines, the Bellman recipe worked by hand (Q=6.5 with two die outcomes), Bellman optimality
(max over arrows), why the best map may tie but V* never does, only the mean reward enters the
recipe, γ flipping the winner between now-big and later-endless (γ=0/0.5/0.9 all worked),
V as average of Q over the map, and the bandit as a one-spot MDP with the equal-arm algebra.
Part 3 -- Grading a Plan by Pencil -- A Table of Lies That Heals Into the Truth
post 34/36 . 2026-06-19 . part 3 of 5 . reinforcement-learning, policy-evaluation, bellman, by-hand
Policy evaluation built from a drone over enemy ground -- a runway (+5), dirt (-2), fuel (-0.1
a move), and a fixed plan of arrows. Grading the plan = filling in the WORTH of every spot
(discounted points collected from here on, obeying the arrows). The one law: a ground spot's
worth is its painted points; an air spot's worth is -0.1 + 0.9*worth(next). The circle: a
worth leans on a neighbour's worth, so there is no one-shot formula. The fix: start every spot
at a lie of 0 and lay the law down again and again (a WALK). Why the lie heals, digit by digit:
the "+" lays a solid -0.1 layer every pass, the painted ground is an anchor, and the truth
climbs one spot per walk (0 -> -0.1 -> -0.19 -> 3.86, frozen). The hard case -- a ground-less
forever-bounce A<->B -- still settles, to -1, and the geometric series 1+0.9+0.9^2+...=10 is
derived from scratch to prove WHY the discount exists. delta/theta stop rule; Gauss-Seidel vs
Jacobi update order; 27-spot 3x3x3 grid worked. Assumes zero background.
Part 4 -- Finding the Best Plan by Pencil -- How Good Arrows Spread From the Runway
post 35/36 . 2026-06-19 . part 4 of 5 . reinforcement-learning, policy-iteration, policy-improvement, by-hand
Policy iteration: no plan given -- finding the best arrows IS the job. The IMPROVE move: at
every air spot, peek all six nudges with the air-law and re-aim the arrow at the biggest peek
(the textbook's Q(s,a) and argmax). The loop: seed any dumb plan, grade, improve, repeat until
frozen. Worked by hand: seed always-Left (air settles to -1), improve -> D scores 4.4, regrade,
improve -> E scores 3.86; good arrows spread one ring per pass in this toy. The proof a frozen
plan is OPTIMAL (Policy Improvement Theorem): improve never worsens; finitely many plans; a
frozen plan is its own best reply -- a fixed point that cannot be beaten. Python: six peeks
hard-coded, the one-ring spread, then the full grade-improve loop. Assumes zero background.
Part 5 -- No Map in the Arguments: Value Iteration Built by Pencil
post 36/36 . 2026-06-22 . part 5 of 5 . reinforcement-learning, value-iteration, dynamic-programming, bellman, by-hand
The third DP algorithm -- the one that needs no plan during the computation. From the same
three scores (GRADE averages by the plan, IMPROVE takes argmax, VALUE-IT takes max), why
max beats any weighted average proved algebraically: the gap = Σ w_i(max - score_i) ≥ 0.
The parking street (spot 3, price 1) worked from the real die table (score = 2.04); why
the 0-start lie heals (truth climbs one horizon per pass). The algorithm in plain words:
apply VALUE-IT at every spot, repeat until delta < theta, extract the plan with one argmax
pass at the end. Code: price 1 fully unrolled, bellman_optimality_update shown, all three
DP functions compared by argument list (pi present = GRADE or IMPROVE; no pi = VALUE-IT).
Value iteration vs policy iteration: same V*, different routes (one max per spot per pass
vs full convergence before each improve). Full Python loop, 23 passes to convergence.
Assumes zero background.
CHAPTER 13 . SAMPLE-BASED LEARNING -- learning a worth from one sample, no die
-----------------------------------------------------------------------------
Part 1 -- No Die, Just a Sample: TD(0) Built by Pencil
post 37/40 . 2026-06-24 . part 1 of 4 . reinforcement-learning, temporal-difference, td-zero, policy-evaluation, by-hand
Chapter 12 graded a plan WITH the die -- averaging the drone recipe over every landing
at once. Take the die away and each move hands back one landing and one reward, no table
of chances. TD(0) folds that single sample in with the bandit nudge (new = old + size x
(target - old)) and lets many small crawls do the averaging the die used to do. You nudge
the worth of the spot you LEFT, toward target = reward + dilute x worth(landing) -- a
guess leaning on a guess, which heals because every sample carries a real reward. Both
cases worked by hand on Cliff Walking (4x12, 48 spots): a mid-game move (target -0.01,
new worth -0.001) and the anchor where the game ends (target = reward alone, the bare
-1 -> new worth -0.1). Three real lab failures shown: forgetting the "=" that stores back (the
table stays a sheet of zeros), nudging the landed spot instead of the left one, and
keeping the discount piece at the anchor. Assumes zero background.
Part 2 -- Max or Honest Average: Q-Learning and Expected Sarsa by Pencil
post 38/40 . 2026-06-24 . part 2 of 4 . reinforcement-learning, q-learning, expected-sarsa, control, by-hand
Part 1 graded a fixed plan; now nobody hands you the arrows, so you must FIND the best
move at each spot -- which forces a worth per (spot, move), a table q, not a row. Pick
mostly the biggest cell (greedy), wander 10% (epsilon). Fix the cell of the move you just
made by the bandit crawl, toward reward + dilute x (future from the landing). That future
is the only fork: Q-learning takes the MAX of the landing's row (dreams you play best);
Expected Sarsa takes the AVERAGE weighted by pick-odds (honest you wander) -- worked by
hand to 0.02 for one and 0.0185 for the other on the same cell. The anchor (game over)
drops the future piece for both. Three lab failures: nudging the new spot, leaking the
just-picked move into the target, keeping the discount at the anchor.
Part 3 -- A Notebook and a Rehearsal: Dyna-Q by Pencil
post 39/40 . 2026-06-24 . part 3 of 4 . reinforcement-learning, dyna-q, planning, model-based, by-hand
Q-learning fixes one worth per real footstep. Dyna-Q keeps a second paper -- a NOTEBOOK
(the model), a page per spot and a line per move, each line holding the (landing, reward)
that move once produced (the two-layer dict drawn in full, with the blank-page guard).
After every real move it REHEARSES: grab a remembered move, read its stored pair, and run
the exact same crawl -- a free correction fed by memory instead of the world. The prize,
worked by hand: a reward spreads BACKWARD along remembered moves with no footsteps -- the
goal's +1 reaching spot 0 through two rehearsals, zero walking. One real footstep, six
beats: correct, record, rehearse, pick, slide, hand back. A rehearsed move that ended the
game drops the future piece (the Part 1 anchor).
Part 4 -- A Clock for Curiosity: Dyna-Q+ by Pencil
post 40/40 . 2026-06-24 . part 4 of 4 . reinforcement-learning, dyna-q-plus, exploration, changing-world, by-hand
Dyna-Q stops exploring once it trusts a path, so it misses a shortcut that opens LATER.
Dyna-Q+ adds a CLOCK -- one number per (spot, move), ticking up every turn, reset to 0 on
the move actually walked -- so staleness is measured. During a rehearsal only, a recalled
move earns a fake bonus kappa x sqrt(tau) on its stored reward (tau 9, kappa 0.001 ->
0.003; tau 2500 -> 0.05), so the longer a move sits untried the more its worth is pumped,
until the chooser re-tries it for real and finds the new shortcut. So even never-walked
moves can be tempted, the first visit to a spot records all four arrows -- the real one
plus three back-to-self placeholders. The bonus belongs to the imagination only; a real
move's reward stays untouched. Ends with the whole chapter in five lines and the honest
model-based-vs-model-free tradeoff.
Reference -- The Sample-based Map: MC, TD, DP, and the Sarsa Family
reference . 2026-06-24 . companion to Chapter 13 . monte-carlo, td, dp, sarsa, q-learning, by-hand
The flip-to decoder for the ideas Chapter 13 leans on -- the ones the quizzes tangle.
Prediction (grade a fixed rule, fill worths) vs control (rewrite the arrows). The
MC/TD/DP triangle split by two questions: does the target LEAN on a guessed next-worth
(MC no, TD and DP yes), and does it use ALL branches via a model (DP) or the ONE real
landing (TD). The Sarsa family -- same nudge, differing only in the PEEK at the landing:
Sarsa = the move actually taken next (on-policy), Q-learning = max (off-policy greedy),
Expected Sarsa = the odds-weighted average -- all three collapsing to reward-alone at the
goal. Plus the trade-offs straight from each target (MC high-variance/episodic/offline vs
TD low-variance/continuing/online), the TD-error terminal-vs-non-terminal, and a full
TD(0) trace by hand to V(A)=0.5, V(B)=0. Adds plain on-policy Sarsa, which Part 2 left out.
CHAPTER 14 . FUNCTION APPROXIMATION -- when there are too many spots for a table
--------------------------------------------------------------------------------
Part 1 -- Too Many Spots for a Table: State Aggregation by Pencil
post 41/41 . 2026-06-24 . part 1 of 4 . reinforcement-learning, function-approximation, state-aggregation, semi-gradient-td, by-hand
Every RL post so far kept one worth per spot in a table -- which dies when a world has
500 spots (or millions), and learns each spot alone. So GROUP the spots: 500 spots into
10 groups of 50, ONE weight per group, and every spot in a group moves together (the
speed-up the textbook calls generalization). A spot's feature is a one-hot pointing at
its group; its worth is weights . feature -- just that group's weight, no table at all.
The weights learn by the exact TD(0) nudge from Chapter 13, the one-hot steering the
whole nudge onto the single group-weight of the spot you left (worked by hand: how-wrong
12.4, w0 -1.5 -> -0.26, the other nine untouched). Then the one genuinely subtle thing --
why it is called SEMI-gradient: the target ALSO rides on the weights, yet we tweak only
the worth(left) half of the slope and throw the target half away (here -1.116 on w2),
treating the target as a fixed label because that is simpler and steadier. Ends with all
500 worths read out at once as runs of duplicates, hard-coded code, and the coarse-fast
vs fine-exact-but-slow trade. Assumes zero background.
Part 2 -- Sharp From Blurry: Tile Coding by Pencil
post 42/42 . 2026-06-30 . part 2 of 4 . reinforcement-learning, function-approximation, tile-coding, coarse-coding, by-hand
Part 1's one-hot forced an either/or: coarse groups share but blur, fine groups sharpen
but stop sharing. The fix on the car-in-a-valley toy, whose state is two real dials
(position, velocity) a table cannot hold: lay the same COARSE ruler down several times,
each shifted. A point lights one tile per ruler -- a k-hot, several switches on, not one --
and its worth is the SUM of the active weights. Worked on one line with two half-width
rulers offset by 0.25: the shifted walls carve quarter-unit cells out of half-unit tiles
(sharp from blurry), and two states share exactly the tiles they have in common -- full
for 0.3 vs 0.4, half for 0.6, none for 0.9, a smooth ramp where Part 1 had a cliff. The
same TD nudge learns, the k-hot steering it onto the active tiles, the step split size =
alpha / tilings so the several tiles sum to one step (how-wrong 0.5, two tiles +0.05 each,
neighbours rising by shared-tile count). Ends with the real Mountain Car -- 2-D square
tiles, eight hashed grids, the eight-camera picture -- and hard-coded code. Control (one
tile-block per move) is Part 3. Assumes zero background.
Part 3 -- Letting the Car Choose: Sarsa Control by Pencil
post 43/43 . 2026-06-30 . part 3 of 4 . reinforcement-learning, function-approximation, sarsa, control, tile-coding, by-hand
Part 2's tiles gave a worth for a STATE, but the car must rank its three pushes (left,
coast, right) -- so give every push its OWN row of weights: worth(spot, move) is the sum
of that spot's active tiles read along the move's row. The car pushes the greedy best but
wanders epsilon of the time so no move goes stale. The Chapter 13 nudge returns, now
grading each push against the move it will ACTUALLY make next: target = reward + dilute x
worth(next spot, next move), and only the taken move's active tiles move. Worked footstep
by hand on the Part 2 toy: at 0.3 push right, fined -1, land 0.6, next push right ->
how-wrong -1.32, the two right-row tiles slide -0.5->-0.632 and -0.8->-0.932, the lesson
spreading across nearby spots but staying inside that move. That "move it really makes" is
what makes it on-policy SARSA; one swap to the max gives Q-learning, one to the odds-
weighted average gives Expected Sarsa. The flag has no next move (the anchor: target =
reward alone), and the -1-per-step fine bends the greedy pick into pumping the swing.
agent_start/step/end and hard-coded code. Assumes zero background.
Part 4 -- Two Stages and a Bend: A Q-Network by Pencil
post 44/44 . 2026-06-30 . part 4 of 4 . reinforcement-learning, function-approximation, neural-network, expected-sarsa, deep-rl, by-hand
Parts 1-3 kept the worth a sum over FIXED features you laid by hand. Across the lander's
EIGHT dials that traps you both ways -- join the dials and the tiles explode (12^8), split
them and the worth goes blind to joint conditions. So stack two straight stages and let
the MIDDLE row LEARN its own features:
psi = sA + b1, then Q = (middle)C + b2. But two straight stages collapse to one (Q = sD +
e with D = AC, shown by hand), so put a BEND between -- flatten negatives to zero, the
ReLU x = max(psi,0) -- which no single sheet can mimic. Worked on a 2->3->4 toy: s=[1,2]
-> psi=[2.5,-0.5,-0.5] -> x=[2.5,0,0] (two middles dead) -> Q=[5.5,-1.5,0,2], four engine-
worths conjured from a dozen dials, no table. Grade the fired engine against the Expected-
Sarsa target y = reward + dilute x (chance-weighted next worths) = 4.42, so how-wrong =
4.42 - 5.5 = -1.08, a row zero but for the fired engine: [-1.08,0,0,0]. Ends on the sting
that opens Chapter 15: one number must correct ~1,300 dials with no feature to point the
way. Forward pass and target as hard-coded code. Assumes zero background.
CHAPTER 15 . TRAINING THE Q-NETWORK -- how one miss teaches a thousand dials
---------------------------------------------------------------------------
Part 1 -- One Miss, a Thousand Nudges: Backpropagation by Pencil
post 45/45 . 2026-06-30 . part 1 of 5 . reinforcement-learning, deep-rl, backpropagation, neural-network, by-hand
Chapter 14 closed on a sting: the net guessed an engine worth 5.5, the target said 4.42,
the miss was one number -1.08 -- and it must fix ~1,300 dials with no feature to point the
way. This post walks that one number backward to every dial by pencil. Ask each dial its
PULL on the fired worth (how much raising it raises that worth) and nudge by size x how-
wrong x pull -- Part 1's fair-share rule with "feature" swapped for "pull." Chase the pull
back through the 2->3->4 toy of Chapter 14: it starts as a single 1 in the fired engine's
slot; sheet C's pull is x dropped into that ONE column (the other three engines sleep, dead
middle rows zero); the bend becomes a gate open only where psi>0, killing the blame headed
for dead numbers ([2,1,-1] x [1,0,0] = [2,0,0]); sheet A's pull is input s times the
surviving blame ([[2,0,0],[4,0,0]]). Every stage's grad is the same shape -- input times
signal. Spend the miss at step 0.01: five of 25 dials move, the biggest to the hardest
pusher, and the fired worth slides 5.5 -> 5.166 toward 4.42. The zeros are the lesson.
Hard-coded forward-and-back code. Assumes zero background.
Part 2 -- A Smarter Step: Adam and Replay by Pencil
post 46/46 . 2026-06-30 . part 2 of 5 . reinforcement-learning, deep-rl, adam, momentum, experience-replay, by-hand
Part 1's nudge was right but spent crudely -- a flat 0.01 step for every dial, and each
move's experience used once in order. Adam fixes the STEP with two per-dial memories: m,
the average of recent pulls (so a dial pulled the same way builds speed, one yanked both
ways cancels), and v, the average of squared pulls (so dividing by sqrt(v) makes big- and
small-pull dials move at one pace), both bias-corrected by 1 - beta^t because they start at
zero -> dial -= alpha x m-hat/sqrt(v-hat). Worked by hand: a steady dial (+0.2,+0.2) keeps
the full 0.1 stride, a flip-flop dial (+0.2,-0.2) shrinks to 0.005. The replay BIN fixes
the DATA: store every (s,a,r,s'), then learn from a RANDOM batch drawn out of it -- breaking
the streak that makes the net forget, and squeezing many lessons from one footstep, each
draw's target rebuilt with the current net (Chapter 13's Dyna-Q rehearsal, now on real
stored moves). Ends with the whole deep-RL loop -- read, pick, act, store, grab, backprop,
Adam step, slide -- and hard-coded code. Assumes zero background.
Part 3 -- The Frozen Twin: A Batch of Misses at Once
post 47/51 . 2026-07-01 . part 3 of 5 . reinforcement-learning, deep-rl, expected-sarsa, target-network, td-error, by-hand
Part 2 grabbed a random batch from the bin and waved at two things: how the batch's miss is
computed all at once, and which net computes it. Both, worked by hand on a batch of two lander
moves. The FROZEN twin reads every landing s' into a worth-row per record; softmax turns each
row into chances (e^0/e^2 worked, [0.096,0.711,0.096,0.096]); chances-dotted-into-worths gives
the Expected-Sarsa landing worth (1.42 and 4.0); times (1 - done) zeroes the crashed record
(else its dead placeholder leaks +2.6 and a crash scores as a reward); target = r + dilute x
that = [3.28, -1.0]. The LIVE net reads every left-spot and -- since the move taken is stored --
picks JUST that column as the guess [2.5, 0.5]. Subtract straight across: the whole batch's
how-wrong in one sweep, [0.78, -1.5], each riding Part 1's backprop into Part 2's one Adam step.
Then why TWO nets: across the several replay passes after one real move the live net moves every
pass, so a target read off IT would flee as fast as you chase it -- photocopy the net before the
passes, freeze the copy, grade every pass against it, refresh between bursts. Guess with the live
net; grade with the frozen twin. Hard-coded code. Assumes zero background.
Part 4 -- The World Calls Three Times: Wiring the Agent by Pencil
post 48/51 . 2026-07-01 . part 4 of 5 . reinforcement-learning, deep-rl, agent, experience-replay, terminal, by-hand
Parts 1-3 built the machine that TRAINS a Q-network -- backprop, Adam, replay, the frozen twin --
but a pile of functions is not an agent. The world runs the loop and CALLS the agent at three kinds
of moment: agent_start (first spot, no reward -- pick a move by softmax over the net's worths and
remember last_spot and last_move), agent_step (every middle moment -- store the 5-tuple
(last_spot, last_move, reward, terminal 0, new spot), run the replay learn from Parts 1-3, pick the
next move, remember it), agent_end (run over -- store the same 5-tuple but terminal 1 with a dead
placeholder spot, learn once more, return nothing). The by-pencil gem: last_move must always hold
the move whose reward the NEXT call receives, or a reward gets pinned on the wrong move (store
(S1,A0,...) when A1 caused R2) -- the two "remember" lines guarantee the right blame. Terminal 1
makes the ended move's target the bare reward via Part 3's (1 - terminal). Strung together over a
few hundred runs, the three calls are a lander that learns to fly. Assumes zero background.
Part 5 -- From Eight Dials to a Soft Landing: The Whole Agent by Pencil
post 49/52 . 2026-07-02 . part 5 of 5 . reinforcement-learning, deep-rl, expected-sarsa, adam, experience-replay, by-hand
Parts 1-4 forged the pieces -- backprop, Adam, replay, the frozen twin, the three agent calls --
each alone. This capstone lights them all at once, following ONE instant of a falling rocket (eight
state dials) the whole way to a nudge of the dials, then the same loop 300 times to a landing. No
Python: every piece rebuilt from nothing in plain pencil, one 2->3->4 toy carried end to end. Score
four engines through two sheets and a bend (s=[1,2] -> Q=[5.5,-1.5,0,2]); ROLL an engine by softmax
at temperature t (near-greedy at t=0.001), never grab; file the world's answer as a five-part note
in a notebook and learn later from random batches; grade a note against a FROZEN twin (worth of the
next spot = chance-weighted average of the twin's scores = 3.8, target = 1 + 0.9x3.8 = 4.42); the
miss (4.42 - 5.5 = -1.08) lands in ONE column; walk it back to every dial by its pull; step not flat
but with Adam, whose first move is a size-blind +/- the step (grade -2.7 -> step -0.001). A handful
of replay passes per move, all reading the same frozen twin; wired into agent_start/step/end; 300
times, the pod learns to land. The whole clock, ticking. Assumes zero background.
CHAPTER 16 . POLICY GRADIENT -- learning the policy itself, not a worth
----------------------------------------------------------------------
Propose and Grade: Actor-Critic by Pencil
post 50/52 . 2026-06-30 . Chapter 16 . reinforcement-learning, policy-gradient, actor-critic, softmax, average-reward, by-hand
Every machine so far learned a WORTH and grabbed the biggest -- which needs a max (bad for
a real-valued move like a steering angle), jolts when two worths cross, and can never say a
MIXED best policy. So learn the policy directly. Keep a PREFERENCE per move (a sum of
weights at the active tiles, Chapter 14) and softmax preferences into chances ([-1,1,2] ->
[0.0351, 0.2595, 0.7054]); SAMPLE from those, never argmax. To judge a taken move, keep a
CRITIC -- a worth per spot as a baseline -- and on the never-ending pendulum measure each
step by the DIFFERENTIAL TD error how-wrong = reward - R-bar + V(landing) - V(left) (worked
to 0.6). That one number drives three crawls: R-bar toward the reward, the critic's worth
toward how-wrong, and each move's preference by size x how-wrong x (took-it? - pi) -- shoving
the taken move UP by (1 - its chance) and the rest down by their odds, so a good surprise
makes its cause more likely (right's preference 2.0 -> 2.035). Actor proposes, critic grades,
both learn from the same gap. Hard-coded code. Assumes zero background.
CHAPTER 17 . TWO THE SAMPLE-BASED MAP SKIPPED -- the whole-return method and planning by surprise
------------------------------------------------------------------------------------------------
Part 1 -- Wait for the Whole Trip: Monte Carlo by Pencil
post 51/52 . 2026-07-01 . part 1 of 2 . reinforcement-learning, monte-carlo, prediction, return, episodic, by-hand
Chapter 13's TD graded the spot you left with one reward plus the GUESSED worth of the spot you
landed on -- a guess leaning on a guess. Monte Carlo leans on nothing: wait for the whole run
(the episode) to END, then grade each spot by its RETURN -- every reward that followed, faded 0.9
per step, totalled backward as reward + 0.9 x next-return. Worked on a courier run S->A->B->END
(returns 6.04, 5.6, 4), then a second run S->A->END (5.5, 5) that averages the samples to
V(S)=5.77, V(A)=5.3, V(B)=4. That averaging is Chapter 12's running-average nudge new = old +
(1/count) x (return - old), fed a whole return instead of one payout -- a target with no guess in
it. The price of leaning on nothing: the run must end (episodic only), the long total is noisy
(many runs to settle), it learns offline at the end. TD walks one step then leans; Monte Carlo
walks all the way and never leans; the rest of the family lives between. Assumes zero background.
Part 2 -- Fix the Biggest Surprise First: Priority Sweeping by Pencil
post 52/52 . 2026-07-01 . part 2 of 2 . reinforcement-learning, priority-sweeping, planning, model-based, dyna, by-hand
Chapter 13's Dyna rehearsed remembered moves at RANDOM to spread worth with no real footstep, and
wasted most early rehearsals on moves whose landing was still 0 (copying a 0 back teaches nothing).
Priority sweeping spends every rehearsal well: give each spot a PRIORITY -- how far its worth would
jump if rehearsed, | new V - old V | with new V = reward + 0.9 x worth(landing) -- and keep a queue
that hands back the biggest jump first. Pop it, apply the update, then wake its PREDECESSORS (the
spots whose move lands on it). On a four-spot corridor where the goal's +1 was just found, the
sweep pops 4 (worth->1), 3 (0.9), 2 (0.81), 1 (0.729) -- four pops, four real moves, the reward
walked all the way back with none wasted, where random grabbing would spend most picks on no-jumps.
A small floor theta drops jumps too tiny to queue. Effort follows the change. Assumes zero background.
CHAPTER 18 . CHOOSING THE MACHINE -- the map that fits the tool to the world
-----------------------------------------------------------------------------
Part 1 -- Four Questions Before Any Arithmetic: Which Machine for Which World
post 53/53 . 2026-07-02 . part 1 of 1 . reinforcement-learning, episodic, continuing, model-based, prediction, control, by-hand
Six chapters built a shelf of machines, and the first mistake on any fresh problem happens
before arithmetic: reaching for the wrong one. Four questions sort every world -- does the
run END (never-ending kills returns AND faded piles: a steady 1 per step piles to 1/(1-0.9)
= 10, 100 at 0.99, unbounded beyond)? do you HOLD the rulebook (then compute, walk nothing)?
can you LIST the spots (else shared dials: groups, tiles, a network)? GRADE or CHOOSE (worth
per spot vs per move)? The whole shelf laid on one map with a row per world, two fresh worlds
sorted as drills (a never-ending greenhouse -> differential actor-critic; a dice game with
the die table printed on the box -> the grade-improve loop). Then the closing honesty: one
good run is an anecdote, not a grade -- the proof is a box, settings pairs x runs x episodes
(3 x 4 x 30 x 300 = 108,000 totals), fix a pair, average down the runs, read the curve.
Assumes zero background.
CHAPTER 19 . LEARNING BY COPYING -- a human already knows how
-----------------------------------------------------------------------------
Part 1 -- The Diary and the Eight Envelopes: Learning by Copying
post 54/58 . 2026-07-03 . part 1 of 5 . imitation-learning, behavior-cloning, action-chunking, dataset, by-hand
Every machine so far learned from a reward trickle -- but a human already knows how to push
the T-shaped block home, and 25,650 recorded moments of showing beat stumbling. One diary
line = five situation numbers (hand x, y; block x, y; block twist) and the human's
two-number answer (a target point). A live machine is slow, so it answers eight ticks at
once: eight sealed envelopes, 8 x 2 = 16 numbers. That forces the re-cut: every line starts
its own eight-move window, none may cross a game seam, so 206 games yield 25,650 - 206 x 7
= 24,208 question-answer pairs through a safe-list that hops the seams (slot 154 holds line
161). Study reads every line; live play goes blind seven ticks of eight. Assumes zero background.
Part 2 -- The Copying Machine by Pencil
post 55/58 . 2026-07-03 . part 2 of 5 . imitation-learning, behavior-cloning, neural-network, mse, by-hand
Raw table numbers (222.0 next to 3.0) choke newborn dials, so every column is re-ruled:
subtract the middle, divide by the spread -- [222.0, 97.0, 223.0, 381.6, 3.0] becomes
[-0.07, -2.03, -0.37, 1.66, 0.67]. Then the web: 5 -> 256 -> 256 -> 256 -> 16, counted to
137,232 dials, and a tiny 2 -> 3 -> 2 twin runs the whole life by pencil -- forward to
[0.7, 0.7], meter (0.36 + 9.00) / 2 = 4.68, one dial wiggle-priced (pull 0.6 x 0.8 = 0.48)
and turned (1.0 -> 0.952), a squash-dead middle pricing its wires at 0. At scale: handfuls
of 128, 189 per read, 400 reads = 75,600 turns -- and the first meter reading is
pencil-predictable: about 1.0. Measured: 0.9885, falling to 0.128 by turn 60. Assumes zero
background.
Part 3 -- One Question, Two Right Answers: Where Copying Breaks
post 56/58 . 2026-07-03 . part 3 of 5 . imitation-learning, multimodal, mse, mean-collapse, by-hand
The diary can hold TWO right answers to one question -- swing above the block or below,
both recorded, both perfect. Graded by squared misses against both, a single print g costs
(g-1)^2 + (g+1)^2 = 2g^2 + 2: printing a truth costs 4, printing the average 0 -- a number
nobody answered -- costs 2, and the bottom can never reach 0. On the table the average of
(230, 260) and (230, 340) is (230, 300): the block's own centre, a stall. Longer training,
more dials, more reads all aim at the same bottom, because the bottom IS the least-miss
print. The disease is the question shape; the cure is to stop printing answers. Assumes
zero background.
Part 4 -- Teaching the Wind: Flow Matching by Pencil
post 57/58 . 2026-07-03 . part 4 of 5 . imitation-learning, flow-matching, noise, mixing, by-hand
Learn arrows, not answers. Each lesson is manufactured by three rolls -- a truth A =
[0.50, 0.60] from the diary, a noise pull N = [0.10, 0.80] from a spread-1 jar, a dial
tau = 0.4 -- mixed to M = tau A + (1 - tau) N = [0.26, 0.72], with teaching answer the
wind A - N = [0.40, -0.20]: walk the remaining 0.6 of it and land exactly on A. The
machine reads 22 wires (5 question + 16 mixture + tau), prints 16 wind numbers through
141,584 dials, graded by the same subtract-square-average meter. Tau earns its wire: at
(M = 0.5, tau = 0) the honest wind is the undecided -0.5; at tau = 0.9 it commits to 5.0.
First meter reading pencil-called at 1 + 1 = 2; measured 2.1136. Assumes zero background.
Part 5 -- Riding the Wind: From Noise to an Answer
post 58/59 . 2026-07-03 . part 5 of 5 . imitation-learning, flow-matching, euler, sampling, by-hand
Answering time has no truth, so the answer is walked: pull a dust speck from the jar and
take n equal rungs, current = current + (1/n) x wind while tau climbs 0 -> 1. A full
4-rung walk by pencil: [0.10, 0.80] -> [0.20, 0.75] -> [0.275, 0.725] -> [0.325, 0.700]
-> [0.3625, 0.680], de-ruled to the table move (265.0, 359.3). Two specks cure the
two-truths crash: one rides to the above-swing (height 255.6), the other to the
below-swing (342.0) -- the old crash point 298.8 exists only if finished answers are
merged, which the rider never does. Cost: ten web passes per envelope instead of one
(16 vs 160 per game). Exam: frozen dials, 100 fresh games, average of per-game best
coverage. Assumes zero background.
CHAPTER 20 . THE PUSH-T MACHINE . from five numbers to a working wind machine
------------------------------------------------------------------------------
Part 1 -- From Five to Twenty-Two: Why Flow Matching Needs a Clock
post 59/60 . 2026-07-04 . part 1 of 3 . imitation-learning, push-t, flow-matching, architecture, action-chunking, by-hand
A square table, a T-block, a round hand, 25,650 recorded diary lines. The five-wire copying
machine (5 -> 256 -> 256 -> 256 -> 16) reads the situation and prints an 8-move plan directly.
It fails at the fork: when one situation has two equally valid paths (swing above or below),
the MSE ruler forces the machine to print their average -- a third path nobody ever chose,
proved by d/dc E[(c-Y)^2] = 0 -> c = (Y1+Y2)/2, worked with Y1=+0.3, Y2=-0.3 -> c=0.0.
The wind fix stops printing the answer and prints a direction instead, but that direction
depends on where the blend point currently sits (16 numbers) and how far along the walk we
are (tau, 1 number) as well as the table situation (5 numbers): 5+16+1=22. Drop tau and the
machine cannot tell early from late; drop the blend and it cannot tell which noise it started
from. All three blend/landing/MSE claims verified in hard-coded Python. Assumes zero background.
Part 2 -- Wrong Wind Shows Every Dial Which Way to Turn
post 60/60 . 2026-07-04 . part 2 of 3 . imitation-learning, push-t, backprop, forward-pass, adam, training-loop, by-hand
The 141,584 dials start random and print wrong wind. One wrong-wind reading teaches every dial:
a 2->2->2 toy runs a full forward pass (pre-activation 0.31, 0.27; ReLU pass-through; predicted
wind [0.294, -0.074] vs true [0.40, -0.20]; MSE loss 0.013556), then a full backward pass by
chain rule (last-layer slope -0.032860 = -0.106 * 0.31; slope flowing back through open ReLU
gates -0.1266 and -0.0046; first-layer slope -0.0633), then one Adam nudge (m=-0.003286,
v=0.0000010798, bias-corrected step +0.0003 lifting W2[o0,h0] from 0.6 to 0.6003). Why
zero_grad->backward->step order is mandatory. Why a dead ReLU is not permanent. Full training
loop: 400 epochs x 189 batches = 75,600 total nudges, eval every 10,000. Assumes zero background.
Part 3 -- Eight Blind Ticks, Then The Overshoot
post 61/61 . 2026-07-04 . part 3 of 3 . imitation-learning, push-t, evaluation, open-loop, mpc, receding-horizon-control, by-hand
The real run: 400 epochs, 75,599 logged dial-nudges (one shy of the 75,600 estimate -- a
dropped fencepost batch), loss 0.59382 -> 0.17142. 35 videos across 7 freezes: nudge 10,001
flails and misses the block, nudge 70,001 commits to one clean push -- but one clip past the
8-second mark reaches the goal strip cleanly, then a still-firing 8th blind push shoves the
block back out. Worked in clean numbers: a plan built once at tick 0 assumes 5 units of push
per tick; real contact delivers 6.5; the block sits inside a 292-308 goal band from tick 5
through tick 7, then tick 8 fires anyway and lands at 288, 4 units past the band's edge,
because the plan never re-measures. The fix, Receding Horizon Control: use only move 1 of
every 8-move plan, throw the rest away, re-plan from a freshly read table every tick -- gap
shrinks by a fixed 0.8375 factor per tick, never changes sign, never has a fixed final push
left to overshoot with. Closes Chapter 20. Assumes zero background.
CHAPTER 21 . SELF, NET, AND THE CALL THAT DOES NOT LOOP . how one dot stops self.net(state) from recursing
------------------------------------------------------------------------------------------------------------
Part 1 -- One Dot Away From An Infinite Loop
post 62/62 . 2026-07-04 . part 1 of 2 . pytorch, python-internals, object-model, nn-module, by-hand
forward calls self.net(state); self.net was itself built from nn.Sequential, and self is built the
same way -- so why does the call not become forward calling forward forever? Traced from a plain
Adder class with a hand-written __call__ (add5(10) = type(add5).__call__(add5, 10) = 15) up to
nn.Module, which fills that same slot with "run self.forward". self.net is a second object, built
once in __init__, never the same object as self -- so self.net(state) hops sideways to a different
forward, while the one-character slip self(state) hops back into the forward already running.
Proved both ways in code: the real line traces two clean __call__ hops (Policy then Layers) and
returns 15.0; the missing-dot version is let loose and actually crashes with RecursionError. Also:
why self.net is built once in __init__ and never rebuilt per call. Assumes zero background.
CHAPTER 22 . THE FATE OF EVERY NUMBER . one plain number traced into the loss and back to its slot
------------------------------------------------------------------------------------------------------------
Part 1 -- The 90 Comes Back As 89.94
post 63/65 . 2026-07-08 . part 1 of 3 . backpropagation, bias, gradient, sgd, broadcasting, by-hand
Textbooks show the fate of variables; this follows one plain number. A twelve-dial machine
guesses 108, 49, 121, 43 against targets 100, 55, 117, 33 -- loss 54. The free add 90 is
copied twice by the +, fused into 108 (18+90) and 121 (31+90), and can never be un-added --
yet the slot still holds 90, and a wiggle passes the + at rate 1. Slope 4 through one guess,
slope 2 through the other, one slot, so 4+2 = 6, and 90 - 0.01*6 = 89.94. Then the 40, whose
two roads FIGHT (-3 wants it bigger, +5 wants it smaller; verdict +2, so 39.98). Proof by
rerun: put 91 in the slot and the loss climbs 54 -> 60.5. Assumes zero background.
Part 2 -- The 52 Forces The Flip
post 64/65 . 2026-07-08 . part 2 of 3 . backpropagation, weights, transpose, relu, dying-relu, by-hand
The multiplied dials. The 2 in the second room rides in at rate 9 and rate 8 (one unit in
its slot moves the loss 54 -> 142.25), and its pull is 4*9 + 2*8 = 52. Hand the code the
sheets as they sit and it grinds 9*4 + 0*2 = 36 -- it paired the bend-mates, the two numbers
that exited the bend together, instead of the dial's own two jobs. The flip (.T) regroups 9
with 8 and reproduces 52: the hand-built number picks the arrangement, the spoken chain rule
cannot. Then the -2 and the -8, whose shared road through the -7 is dead at the bend (rate
0) -- dials that learn only from their surviving line. Assumes zero background.
Part 3 -- Born, Works, Dies -- Then All Twelve Dials Turn
post 65/65 . 2026-07-08 . part 3 of 3 . backpropagation, sgd, dead-relu, gradient-check, cache, by-hand
Fates of the numbers nobody tweaks: the reading 3 (shapes two squares of the loss, and no
slot ever reaches back -- the machine may not edit the world), the -7 (hands 0*3 and 0*(-1)
forward, and the loudest pull in the run, 15, dies at its bend: 15*0 = 0; after the turn it
drifts to -7.06, deader than before), and the freight pull 5 (born 4*2 + (-3)*1, delivers
10, 5, and 5 to three dials, then is never referenced again). Then all twelve dials turn at
once and the machine reruns: loss 54 -> 30.1145, three guesses closer, one worse -- a
committee verdict. The slopes promise a fall of 52.3 at step 0.01 but deliver 23.9; at step
0.0001 they promise 0.523 and deliver 0.5197 -- slopes tell the truth only near where they
were read. Assumes zero background.
CHAPTER 23 . POLICY GRADIENTS . improving a rule from its own scores, with no answer key anywhere
------------------------------------------------------------------------------------------------------------
Part 1 -- No Answer Key, Only a Score
post 66/69 . 2026-07-09 . part 1 of 4 . reinforcement-learning, policy-gradient, return, reward-to-go, discount, by-hand
The jump from copying an answer key to learning from a bare score. A four-tick run pays
rewards 1,2,3,4 with discount 0.9. The blunt whole-run pile stamps 8.146 on every move; the
fair reward-to-go charges each move only its own future -- 8.146, 7.94, 6.6, 4 -- folded
backward because each score leans on the one after it. Assumes zero background.
Part 2 -- Why the Log Times the Score Turns the Dial
post 67/69 . 2026-07-09 . part 2 of 4 . reinforcement-learning, policy-gradient, reinforce, log-prob, likelihood-ratio, by-hand
Why one downhill step on -log(chance)*score turns the coin toward good moves. A 50/50 coin
takes Left and scores 14.009; the swap "change in p = p times change in log p" turns climbing
the expected score into an average over sampled moves, so the world's own dice differentiate
away and vanish. The dials move to 0.07 and -0.07, and Left's chance climbs 0.500 -> 0.535.
Assumes zero background.
Part 3 -- Subtract a Yardstick, Lose No Truth
post 68/69 . 2026-07-09 . part 3 of 4 . reinforcement-learning, policy-gradient, baseline, advantage, critic, by-hand
Raw score confuses a good move with a good place. Subtract a yardstick V(s): the advantage
score-minus-V splits 14.009 and 12.0 into a clean +1.009 and -1.0. Proof the yardstick cannot
tilt the average push -- the sum of each move's chance times its dial-rate is 1 - 1 = 0,
always. The critic learns V by shrinking a squared miss. Assumes zero background.
Part 4 -- Trust the Critic a Little: GAE by Pencil
post 69/69 . 2026-07-09 . part 4 of 4 . reinforcement-learning, policy-gradient, gae, td-error, bias-variance, by-hand
Two broken advantages -- the full run honest but noisy, the one-step calm but biased -- blended
by a dial. The surprise d = reward + gamma*V(next) - V(here); at the last tick "done" zeroes the
reach past the end. GAE folds the surprises backward by gamma*lambda: lambda 0 gives 1.01 (trust
the critic), lambda 1 gives 1.91 (trust the run), 0.95 gives 1.8458. Closes Chapter 23.
Assumes zero background.
----------------------------------------------------------------------------------------------
the method: ban the jargon . draw it first . solve by pencil . then write the code
built with: pandas, scikit-learn, NumPy, and a pencil . pure HTML+CSS, no JavaScript
follow along: subscribe via RSS . dark mode follows your system setting
. no cookies, no trackers . source on GitHub
==============================================================================================