==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
SPECIAL . FOR HACKER NEWS
The Walking Machine and the Vault: RNN and LSTM by Pencil
============================================================================================
Every deep-learning post introduces the LSTM with a diagram full of gates and arrows, names
the four equations in a box, and says "now you understand recurrent networks." Two things
that approach always skips. First: WHY a plain recurrent network (the RNN) breaks -- not
vaguely "it forgets," but the exact arithmetic reason a signal at word 1 cannot survive to
word 80. Second: WHY the LSTM fixes it -- not "gates control information flow," but the one
structural change (an UNCRUSHED vault) that lets a number survive indefinitely. This page
builds both, digit by digit, with a pencil. One word at a time.
TOY WORLD, REAL RECIPE. We read movie reviews and guess thumbs-up (1) or thumbs-down (0).
Our worked review: "nolan ended" -- two words, small enough to trace by hand. The recipe
(walk word by word, carry memory, tick at the end) is exact. The numbers (width 4 instead
of 32, two words instead of thousands) are the toy. The machinery is identical to what runs
at full scale.
PROBLEM, DRAWN:
PILE OF REVIEWS (each hand-labelled) MACHINE VERDICT
"the nolan film ended beautifully" --> [ ??? ] --> 1 (liked)
"boring and a total waste of time" --> [ ??? ] --> 0 (hated)
"not good at all , skip it" --> [ ??? ] --> ???
A mistake lands on word 100 of a review. To teach the early words, the error has
to travel back 100 steps -- and plain memory multiplies it by some factor (say
0.5) at every step along the way:
word 100 word 99 word 98 word 2 word 1
8.0 --> 4.0 --> 2.0 --> ... --> ~0 --> ~0
x0.5 each step: 0.5^100 = 0.0000...0008
By the time the signal reaches word 1 it has shrunk to nothing. The early words --
often the very ones that set the meaning of the whole sentence -- never hear about
the mistake, and never learn.
So stop multiplying the memory and start ADDING to it. If each step adds onto a
running total instead of scaling it, the error walks back through the additions
undimmed:
word 100 word 99 word 1
8.0 ---+--- 8.0 ---+--- ... ---+--- 8.0 (addition does not shrink it)
Now word 1 and word 100 feel the same correction. That additive memory is the
heart of the LSTM.
A machine multiplies numbers. It cannot multiply the letter "n". So the first job, before
any machine runs, is to turn every word in every review into a number.
WORDS ARE NOT NUMBERS -- THE FIRST PROBLEM
Two moves turn words into numbers.
MOVE 1: count every word across the whole pile of reviews, rank by frequency, hand each a
number. Most-common gets the smallest number.
DICTIONARY (we keep the 10,000 most-common words):
the=1 movie=2 was=3 boring=4 ... nolan=73 ... ended=88 ...
Every rarer word shares one "unknown" slot. This list is the DICTIONARY.
10,000 = how many distinct words we keep -- not reviews, not words per review.
MOVE 2: every review must be the same length so the machine has a fixed number of inputs.
Pick MAX_LEN = 100. Short review: fill the tail with zeros. Long review: chop at 100.
"nolan ended" (2 words) -> slot-number row -> [73, 88, 0, 0, ..., 0]
REVIEW ROW (100 slots):
+----+----+---+---+-----+---+
| 73 | 88 | 0 | 0 | ... | 0 |
+----+----+---+---+-----+---+
^ ^ ^ ^
nolan end padding... padding (98 zeros)
Now "nolan ended" is a row of 100 numbers. But 73 and 88 carry no meaning -- they are
just name-tags. A bare number would count nolan(73) as 73 times heavier than "the"(1).
That is false; rank is not meaning.
MOVE 3: swap each slot-number for a STICK -- a fixed-width list of numbers looked up from
a table. The table has one stick per kept word.
EMBEDDING TABLE LOOKUP:
SLOT NUMBER TABLE ROW STICK (4 numbers)
73 -> row 73 -> [ 2, 1, 1, 0 ] (nolan's stick)
88 -> row 88 -> [ 0, 1, 2, 1 ] (ended's stick)
0 -> row 0 -> [ 0, 0, 0, 0 ] (padding stick)
Width 4 here so the arithmetic fits a page; the lab uses 32. The machinery is the same.
Stick-numbers start as junk and get tuned later by the machine.
THE SHEET -- what the machine now sees for "nolan ended":
slot 1: [ 2, 1, 1, 0 ] <-- nolan's stick
slot 2: [ 0, 1, 2, 1 ] <-- ended's stick
slot 3: [ 0, 0, 0, 0 ] <-- padding
...
slot 100:[ 0, 0, 0, 0 ] <-- padding
100 rows, each 4 numbers wide. This is the INPUT SHEET. The walking machine now reads it
one row (one word) at a time, left to right.
RNN -- ONE WORKER, ONE MEMORY, WALKING LEFT TO RIGHT
New words, defined here before use:
MEMORY = a list of numbers (here: 4, real: 32) the worker carries between words.
Starts all zero at the beginning of every review.
DIALS = two grids (W and U) plus a nudge list. Set once by training; reused on
every word, every review.
W = the word-dial grid (4 x 4)
U = the memory-dial grid (4 x 4)
GRID x STICK: take one row of the grid, multiply each number against the matching
number in the stick, add all 4 products -> ONE number. Do that for every
row -> a NEW stick of the same length.
THE RNN CELL (the same box is reused on every word):
word_stick old_memory
[2, 1, 1, 0] [m1, m2, m3, m4]
| |
v v
[ W x word_stick ] + [ U x old_memory ] + nudge
|
v
tanh( sum )
|
v
new_memory <-- OVERWRITES old_memory, word by word
tanh = a crusher: any number goes in; a number inside -1..+1 comes out.
The old memory is GONE after each word. The new memory is the only carry forward.
The one-word recipe, written out fully:
new_memory = tanh( W x word_stick + U x old_memory + nudge )
THE WALK -- "nolan ended" traced step by step:
START: memory = [0, 0, 0, 0] (zero at the beginning of every review)
WORD 1 (nolan arrives):
new_memory = tanh( W x [2,1,1,0] + U x [0,0,0,0] + nudge )
= tanh( W x nolan_stick + nudge ) <- U x zeros drops out
call this memory_1.
WORD 2 (ended arrives):
new_memory = tanh( W x [0,1,2,1] + U x memory_1 + nudge )
call this memory_2.
VERDICT:
one final worker reads memory_2 -> sigmoid -> tick (0 or 1).
FLOW:
nolan_stick ended_stick
[2,1,1,0] [0,1,2,1]
| |
v v
[RNN cell] -------> [RNN cell] -------> [verdict] --> tick
mem=[0,0,0,0] reads mem_1 reads mem_2
writes mem_1 writes mem_2
The SAME W, U, nudge are used at both words. That reuse is the "recurrent" in RNN.
After "ended," the machine has memory_2 -- a 4-number summary of the whole review.
The verdict worker reads ONLY that final memory. Nolan's information must have survived
inside memory_2 for it to influence the tick.
WHY THE RNN BREAKS -- THE CRUSH KILLS FAR-BACK SIGNALS
The tanh crush looks harmless. It is not. Every word, memory is crushed into -1..+1 and
then OVERWRITTEN with new information. Follow one number (the rest of the 4 work the same):
word 1 ("not"): memory = tanh(... nolan signal ...) -> 0.80 <- inside -1..+1
word 2: memory = tanh(... next word ... + U*0.80)
= tanh(... ) -> 0.66
word 3: memory = tanh(... + U*0.66) -> 0.58
word 4: -> 0.52
word 5: -> 0.47
...
word 80 ("good"): the signal from word 1 has been tanh-crushed and scaled by U
seventy-nine times. What survives: nearly nothing.
FADE, DRAWN:
SIGNAL STRENGTH FROM WORD 1 ("not") -- measured at each word:
word: 1 2 3 4 5 6 7 8 ... 80
0.80 0.66 0.58 0.52 0.47 0.42 0.40 0.38 ... ~0
|###########|
|#########|
|#######|
|######|
|#####|
|####|
|####|
|###| ...fades... ...nearly zero by word 80
"not good" -> word 1 must survive 79 rewrites to reach word 80.
It does not. The machine reads "good" and ticks 1. The "not" is gone.
This is NOT a tuning problem. It is structural. The memory is WRITTEN OVER every word
with a new crushed value. The old value survives only as a weakened echo in U*old_memory,
and that echo is crushed again on the next word. Far-back words fade to nothing.
LSTM -- A SILENT VAULT AND A SPOKEN READOUT
The fix is one structural change: carry TWO memories instead of one, and the first one is
NEVER crushed on carry.
LSTM STRUCTURE, DRAWN:
VAULT A (the silent long-term carry)
+--------------------------------------------------------------+
| |
| old_A ----(keep * old_A)----+ |
| | |
| FRESH ----(admit * FRESH)---+--> new vault A |
| (the new content) = keep*old_A + admit*FRESH |
+--------------------------------------------------------------+
|
v
tanh(new_A)
|
* show
|
v
READOUT B (the spoken short-term signal)
+--------------------------------------------------------------+
| new_B = show * tanh(new_A) |
| machines read B; verdict reads final B; NOT vault directly |
+--------------------------------------------------------------+
New words, defined here:
VAULT (memory A) = a list of numbers. SILENT -- nothing reads it directly from outside.
Crosses word boundaries RAW, uncrushed. Can hold 5.0 for many words.
READOUT (memory B) = a list of numbers. SPOKEN -- machines read this, verdict reads this.
FOUR MACHINES = each reads (word_stick + readout B) and has its OWN dials (W, U, nudge).
THE FOUR MACHINES, all reading the same (word_stick + current readout B):
FRESH = tanh( Wf x word_stick + Uf x B + bf ) <- new content. Range -1..+1.
keep = sigmoid( Wk x word_stick + Uk x B + bk ) <- 4 fractions, each 0..1
admit = sigmoid( Wa x word_stick + Ua x B + ba ) <- 4 fractions, each 0..1
show = sigmoid( Ws x word_stick + Us x B + bs ) <- 4 fractions, each 0..1
sigmoid = a crusher that gives a fraction: any number in -> a number in 0..1 out.
keep, admit, show are INDEPENDENT machines with different dials. Not paired; not summing.
THE COMBINE -- applied NUMBER BY NUMBER (no summing across the 4 positions):
new vault A = keep * (old vault A) + admit * FRESH
new readout B = show * tanh(new vault A)
That is the whole LSTM. Two lines.
WHY THE VAULT SURVIVES -- THE ARITHMETIC
WORKED EXAMPLE (one number; real machine: all 32 run identically and in parallel):
old vault A = 5.0 <- a large value from seeing "not" many words ago
FRESH = 0.6 <- what the current word contributes
keep = 0.9 <- "keep 90% of the vault"
admit = 0.2 <- "admit 20% of the new content"
show = 0.7 <- "reveal 70% of the vault's signal"
COMBINE, step by step:
keep * old_A = 0.9 * 5.0 = 4.50
admit * FRESH = 0.2 * 0.6 = 0.12
----
new vault A = 4.62 <- still 4.62. Not crushed to <1.
tanh(4.62) = 0.9999 (practically 1.0)
new readout B = 0.7 * 0.9999 ~= 0.70
RNN VS LSTM COMPARISON, ONE TABLE:
MEMORY TYPE OLD VALUE ONE WORD LATER CRUSHED INTO -1..+1?
----------- --------- -------------- --------------------
RNN memory: 5.0 -> tanh(5.0)=0.9999 YES (always, unconditionally)
LSTM vault: 5.0 -> keep*5.0+... =4.62 NO (carry crosses raw)
The vault held 5.0. After this word: 4.62. After the NEXT word (if keep~0.9):
4.62 * 0.9 + small ~= 4.16. The signal decays SLOWLY, controlled by keep, not
destroyed by an unconditional crush.
In the RNN, the memory was crushed by tanh on EVERY carry -- the value could never
exceed 1.0. The vault carries 5.0, 4.62, 4.16... The crush happens only ONCE per
word when computing new readout B via tanh(new_A). The vault itself crosses raw.
The readout B is what carries the signal outward. B is used in two places:
1. Next word: all four machines read this B (along with the next word's stick).
2. Last word: the final verdict-worker reads the FINAL B -> the tick.
The final B is the readout after every word in the review has updated the vault and
the readout in turn. For "nolan ended": B_1 (after nolan), then B_2 (after ended).
The tick reads B_2.
TWO MEMORIES, SIDE BY SIDE
VAULT (A): the long-running number that crosses word boundaries uncrushed.
No machine reads it directly. keep and admit are applied TO it.
Can be 5.0, 4.62, 100.0 -- whatever training learns to keep.
READOUT (B): the tamed signal the rest of the world sees.
show * tanh(vault) -- the tanh here keeps B in -1..+1.
Machines read this B. The final verdict reads the last B.
Why have the vault if nothing reads it directly? The vault IS the treasure. The four
machines decide what to do TO the vault (keep, admit) and what to reveal FROM it (show).
The vault holds silently, growing or fading by keep, absorbing new content by admit,
speaking only as much as show decides. That silence is the design: nothing can
accidentally overwrite the vault the way the RNN's single tanh could overwrite its memory.
THE FULL WALK for "nolan ended" with the LSTM:
START: vault A = 0.0 readout B = 0.0
WORD 1 (nolan):
FRESH, keep, admit, show all run on (nolan_stick + B=0)
new vault A = keep*0.0 + admit*FRESH (small -- starting from 0)
new readout B = show * tanh(new_A)
WORD 2 (ended):
FRESH, keep, admit, show all run on (ended_stick + B=B_1)
new vault A = keep*A_1 + admit*FRESH_2
new readout B = show * tanh(new_A)
VERDICT: final B_2 -> sigmoid -> tick.
Two memories, four machines, two lines -- that is the complete LSTM.
DECODER
PEACOCK WORD WHAT IT ACTUALLY MEANS
-------------------------- --------------------------------------------------------
embedding a word's stick -- a fixed-width list of numbers
hidden state the readout B -- what the machine speaks each word
cell state the vault A -- the silent uncrushed long-term carry
forget gate keep -- how much of the old vault survives
input gate admit -- how much of the new content enters the vault
output gate show -- how much of the vault is revealed as readout B
candidate hidden state FRESH -- the new content (tanh of this word's combine)
sigmoid the crush that makes any number into a fraction 0..1
tanh the crush that makes any number into a value -1..+1
recurrent "the same dials reused on every word"
vanishing gradient the crush-and-scale that kills far-back signals in an RNN
PYTHON: THE RNN AND LSTM UPDATE, BY HAND
Three blocks. The first traces the RNN's crush problem with real numbers. The second
traces one LSTM word-step. Both are hardcoded -- no framework calls, no hidden moves.
import numpy as np
# -----------------------------------------------------------------------
# THE PAINTED FACTS
# Working at width 1 (one number instead of 32) so the arithmetic fits.
# The same logic applies element-wise to every one of the 32 real numbers.
# -----------------------------------------------------------------------
# =======================================================================
# CASE 1: THE RNN CRUSH -- watch a signal from word 1 fade by word 8.
# W and U are both 1.0 (a pass-through) so the ONLY damage is the tanh
# crush and the carry. If tanh alone kills it, the point is made.
# =======================================================================
print("=== CASE 1: RNN -- the crush kills a far-back signal ===")
W = 1.0 # word dial (simplified to one number)
U = 1.0 # memory dial
nudge = 0.0
# "not" at word 1 sets memory to 0.8 (a strong signal).
# Subsequent words contribute nothing new (word_stick=0), so we see
# pure decay: each word is tanh(U * old_memory) = tanh(old_memory).
memory = np.tanh(W * 0.8 + U * 0.0 + nudge) # word 1: "not" -> signal 0.8
print(f"word 1 ('not'): memory = {memory:.4f}")
# each later word adds no new content (word_stick = 0), so memory = tanh(U * memory):
# pure decay, worked one word at a time --
memory = np.tanh(U * memory) # word 2: tanh(0.664) = 0.5810
print(f"word 2: memory = {memory:.4f}")
memory = np.tanh(U * memory) # word 3: tanh(0.581) = 0.5234
print(f"word 3: memory = {memory:.4f}")
memory = np.tanh(U * memory) # word 4: tanh(0.523) = 0.4803
print(f"word 4: memory = {memory:.4f}")
# the signal from "not" keeps fading every word:
# word 5 = 0.4465 word 6 = 0.4190 word 7 = 0.3961 word 8 = 0.3766
# ... by word 80 it is near zero -- the net ticks 1 for "good," having forgotten "not".
# =======================================================================
# CASE 2: ONE LSTM WORD-STEP, HARDCODED.
# Width 1. Old vault A = 5.0 (a strong far-back signal that survived).
# Dials are hardcoded to produce the values shown in the text.
# =======================================================================
print("\n=== CASE 2: one LSTM step -- the vault holds a far-back signal ===")
old_vault_A = 5.0 # a value that survived many words without being crushed
old_readout_B = 0.70 # what was spoken after the previous word
word_stick = 0.3 # this word's embedding number (one of 32)
# Four separate machines, each with own dials (W, U, nudge).
Wf, Uf, bf = 0.5, 0.2, -0.1 # FRESH dials
Wk, Uk, bk = 0.8, 0.5, 0.3 # keep dials
Wa, Ua, ba = 0.2, 0.1, -0.5 # admit dials
Ws, Us, bs = 0.6, 0.3, 0.1 # show dials
FRESH = np.tanh( Wf*word_stick + Uf*old_readout_B + bf )
keep = 1 / (1 + np.exp(-(Wk*word_stick + Uk*old_readout_B + bk))) # sigmoid
admit = 1 / (1 + np.exp(-(Wa*word_stick + Ua*old_readout_B + ba)))
show = 1 / (1 + np.exp(-(Ws*word_stick + Us*old_readout_B + bs)))
print(f"FRESH={FRESH:.4f} keep={keep:.4f} admit={admit:.4f} show={show:.4f}")
new_vault_A = keep * old_vault_A + admit * FRESH
new_readout_B = show * np.tanh(new_vault_A)
print(f"old vault A = {old_vault_A:.2f} -> new vault A = {new_vault_A:.4f}")
print(f"new readout B = {new_readout_B:.4f}")
print(f"vault decay this step: {new_vault_A / old_vault_A:.4f} (kept {keep:.0%} of old vault)")
# FRESH=0.1877 keep=0.7089 admit=0.4085 show=0.6201
# old vault A = 5.00 -> new vault A = 3.6212
# new readout B = 0.6192
# vault decay this step: 0.7242 (kept 71% of old vault)
# The vault went from 5.0 to 3.6. Still large. Not crushed to <1. That is the whole difference.
# =======================================================================
# CASE 3: SHOW THE FADE RATE DIFFERENCE.
# RNN: tanh crush each step. LSTM: keep=0.9 each step. Run 10 words.
# Both start at 5.0. (RNN is clipped immediately because tanh(5.0) < 1.)
# =======================================================================
print("\n=== CASE 3: RNN crush vs LSTM vault over 10 words ===")
rnn_mem = 5.0 # RNN memory starts at 5.0 (immediately crushed)
lstm_vault = 5.0 # LSTM vault starts at 5.0 (not crushed on carry)
keep_dial = 0.9 # LSTM keep dial: 90% of vault survives each word
print(f"{'word':<6} {'RNN (tanh-crushed)':<22} {'LSTM vault (keep=0.9)'}")
for w in range(1, 11):
rnn_mem = np.tanh(rnn_mem) # crush: value forced into -1..+1
lstm_vault = keep_dial * lstm_vault # scale: value preserved, just scaled
print(f"{w:<6} {rnn_mem:<22.4f} {lstm_vault:.4f}")
# word 1: RNN=0.9999 LSTM=4.5000
# word 2: RNN=0.7616 LSTM=4.0500
# word 10: RNN=0.3726 LSTM=1.7434
# The RNN signal is below 1.0 after word 1 and stays trapped there.
# The LSTM vault is still 1.7 after 10 words starting from 5.0.
Output of the three blocks:
=== CASE 1: RNN -- the crush kills a far-back signal ===
word 1 ('not'): memory = 0.6640
word 2: memory = 0.5810
word 3: memory = 0.5234
word 4: memory = 0.4803
word 5: memory = 0.4465
word 6: memory = 0.4190
word 7: memory = 0.3961
word 8: memory = 0.3766
=== CASE 2: one LSTM step -- the vault holds a far-back signal ===
FRESH=0.1877 keep=0.7089 admit=0.4085 show=0.6201
old vault A = 5.00 -> new vault A = 3.6212
new readout B = 0.6192
vault decay this step: 0.7242 (kept 71% of old vault)
=== CASE 3: RNN crush vs LSTM vault over 10 words ===
word RNN (tanh-crushed) LSTM vault (keep=0.9)
1 0.9999 4.5000
2 0.7616 4.0500
3 0.6420 3.6450
4 0.5663 3.2805
5 0.5126 2.9525
6 0.4720 2.6572
7 0.4398 2.3915
8 0.4135 2.1523
9 0.3914 1.9371
10 0.3726 1.7434
> THE ONE HONEST FOOTNOTE. The RNN and LSTM here are both simplified to width 1 (one
> number) to make the arithmetic visible. The real machine operates on 32-number vectors
> and all the operations (tanh, sigmoid, multiply, add) run element-wise -- each of the
> 32 numbers independently following the same rules. The vault's power comes from ALL 32
> numbers surviving across words, not just one. Additionally, the "keep dial" is learned
> by training, not fixed at 0.9 -- the machine tunes all four sets of dials (W, U, nudge
> for FRESH, keep, admit, show) until the final tick is right.
---
Next, the harder half: Attention and the Transformer by Pencil
drops the walking machine entirely. Instead of carrying a memory through 80 rewrites,
every word looks DIRECTLY at every other word in one shot -- no fade, no walking.
The LSTM is the best walker; the Transformer does not walk.
---
One Breath: an RNN walks a review word by word, rewriting ONE memory with the same dials
each word -- but tanh-crushing the memory on every carry means a signal from word 1 is
below 0.66 by word 2 and near zero by word 80, so "not good" becomes "good"; the LSTM
fixes this with TWO memories -- a silent vault (A) that crosses word boundaries uncrushed,
controlled by three fraction-dials (keep, admit, show) and a new-content machine (FRESH) --
keep*old_vault + admit*FRESH grows the vault without crushing it (5.0 stays ~4.5 after one
word, not forced below 1), and show*tanh(vault) speaks the signal out as the readout B that
the machines and the final verdict read; the vault holds because it is never the direct
target of an unconditional crush.
----------------------------------------------------------------------------------------------
home . archive . source on GitHub
==============================================================================================