==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 10 . MACHINES THAT READ WORDS . PART 1 OF 3
Words Into a Machine: The Notepad and the Walking Worker
============================================================================================
A brand-new pile.
A photo is already numbers: each pixel is a brightness from 0 to 1, ready to feed to a
machine that multiplies numbers.
But this pile is WORDS: movie reviews.
Each review is marked 1 (the writer liked it) or 0 (did not).
The goal is the old goal in new clothes -- read a NEW review, guess 1 or 0.
One yes/no per review.
(The textbook calls this "sentiment": thumbs up or down.)
The wall is right at the door.
A "factory" here means a stack of clerks and dials, where a clerk multiplies its inputs by
dials (tunable numbers) and adds them up.
Such a factory can only multiply NUMBERS.
It cannot multiply the word "boring".
So the whole front of this post is one job: turn words into numbers WITHOUT throwing away
what the words mean.
Then we build the first text-factory -- three floors.
Then we walk one review through the middle floor by hand, one word at a time, until a
running memory falls out the end.
Pencil and scratch paper out. Every number here is recomputed where it is needed.
NEW SHEET, AND THE WALL
The sheet:
text (a movie review, in words) answer
--------------------------------------------------- ------
"complete waste of my evening" 0 (did not like it)
"an absolute masterpiece, loved every minute" 1 (liked it)
... (thousands of reviews)
Each ROW is one review.
The answer column is 1 or 0.
We want a rule that reads the words of a new review and guesses the 1 or 0.
The wall, stated plainly:
photos are ALREADY numbers (pixels 0..1, ready to feed in)
reviews are WORDS ("boring", "masterpiece") -- NOT numbers
a factory of clerks + dials can only multiply NUMBERS, not the word "boring"
So we must number the words first.
Three moves get us there -- a numbering, a length-fix, and a split.
WORDS ARE NOT NUMBERS, SO HAND EACH ONE A NUMBER (Q1, Q2A-B)
Walk every review in the pile and COUNT how often each word appears.
This is the oldest trick in the book -- count every word in a giant pile of text.
Then the most-common word gets the smallest number.
"the" appears most -> 1
"movie" next -> 2
"was" -> 3
"boring" -> 4
...
Now a CHOICE, not a fact: keep only the 10,000 most-common words.
Every rarer word is a typo, a name, or weird slang seen twice in the whole pile.
Each such rarer word is dumped into ONE shared bin we will call [unknown].
We reserve the number 0 for "empty" (the next move needs it), so [unknown] gets its own
number instead of 0.
The counts:
reviews in the pile = maybe 50,000 <- ROWS of the sheet
distinct words KEPT = 10,000 <- a knob WE set (call it VOCAB_SIZE)
every rarer word -> one shared [unknown] number
Why toss the rare words?
A word seen twice in 50,000 reviews cannot teach the machine anything reliable.
Two sightings show no pattern.
The 10,000 is not sacred -- it is a knob (a number we are free to set).
Bigger keeps more vocabulary and costs more dials; smaller is leaner.
Mind the count here: 10,000 is how many distinct WORDS we keep.
It is NOT the number of reviews.
Reviews are the rows (maybe 50,000); kept words are the vocabulary (10,000).
These are two different counts that both happen to be in the tens of thousands.
After this the review "boring movie the" becomes [4, 2, 1] -- three numbers, one per word.
Each number is just a NAME TAG pointing at a word.
A FIXED FACTORY NEEDS ONE LENGTH, SO PAD EVERY REVIEW (Q2C)
IN HAND: every word now has a number (the=1, boring=4, and so on).
We kept 10,000 words; the rest are pooled as [unknown].
A review is now a row of word-numbers of WHATEVER length it was.
Here is the problem.
The factory has a FIXED count of dials.
It cannot eat an 8-number review and a 200-number review with the same dials.
The shapes do not match.
So force every review to exactly the same length.
Call that length MAX_LEN = 100 numbers (a knob, a number we set).
review too LONG (200 words) -> chop everything past 100 (keep the first 100)
review too SHORT (8 words) -> fill the end with 0s up to 100
Worked tiny example. "boring film" -> [4, 17] -- two real numbers. Pad to length 6:
[4, 17, 0, 0, 0, 0] <- 2 real word-numbers, then 4 zeros
The 0 means "empty slot, nothing here."
That is why 0 was reserved and never handed to a real word.
After this EVERY review is a line of exactly 100 numbers.
Now the factory can eat them, all the same shape.
MAX_LEN = 5. The review "the movie was boring" numbers to [1, 2, 3, 4].
What is the padded row? And what does "the the the the the the the" (7 words) become?
check your slate:
[1, 2, 3, 4] has 4 numbers, need 5 -> pad one zero -> [1, 2, 3, 4, 0].
"the" x7 = [1,1,1,1,1,1,1], 7 numbers, need 5 -> chop to first 5 -> [1, 1, 1, 1, 1].
NEVER GRADE ON STUDIED CARDS, SO CUT STUDY FROM EXAM (Q2D)
The honest rule: never grade the machine on cards it studied.
So cut 80% of the reviews to study from.
Then seal the other 20% as the exam.
The exam reviews are not looked at until the very end.
(A fuller setup splits THREE ways -- a study pile, a practice pile to tune knobs, and a
sealed exam. This lab uses just two piles, study and exam. The rule is identical either
way: the graded pile is sealed.)
After all three moves: every review is a line of exactly 100 word-numbers.
The pile is cut into a study 80% and a sealed exam 20%.
The factory can finally be built.
THE FIRST TEXT-FACTORY: THREE FLOORS (Q3)
IN HAND: every review is now 100 word-numbers (e.g. [4, 73, 1, 87, 0, 0, ...]), 0 = empty.
We feed that line of 100 numbers into a factory with three floors:
100 word-numbers in
|
floor 1: NOTEPAD swap each word-number for a row of meaning-numbers
|
floor 2: ONE WORKER read the rows IN ORDER, carry a running memory
|
floor 3: FINAL CLERK read the last memory -> one chance, liked (1) / not (0)
Each floor gets its own section.
Floor 1 fixes the name-tag problem.
Floor 2 is the heart of the chapter.
Floor 3 is a single clerk that ends in an S-curve -- a fixed bend that crushes any number
into a chance between 0 and 1 (more on it at floor 3).
FLOOR 1: THE NOTEPAD (EMBEDDING)
The word-number is just a NAME TAG.
boring=4 and masterpiece=73 -- the size of the number means NOTHING.
Feed it raw to a clerk and masterpiece (73) would count 73 times heavier than "the" (1).
That is pure nonsense.
A name tag is not a measurement.
So instead of feeding the bare number, jot a NOTE about each word.
The note is a little row of real numbers that says what the word is like.
This note of real numbers is what the textbook calls an EMBEDDING.
Start with one number per word, a LEAN:
"boring" appeared in 1000 reviews, 900 of them rated 0 -> note: leans DOWN
"masterpiece" -> note: leans UP
"the" shows up everywhere, no lean -> note: ~middle (useless)
One number per word means a notepad 10,000 rows tall and 1 wide.
That single LEAN number is already an embedding, at width 1.
But one number per word is too thin.
Here is the reason, and it is the whole reason the note is WIDE.
The word "not" does not lean up or down on its own.
Its whole job is to FLIP the next word.
"not boring" is a compliment.
A single lean-number cannot say "I am a flipper."
So a word needs several slots:
"boring" note: lean = down, flipper = no, amplifier = no, ...
"not" note: lean = none, flipper = YES, amplifier = no, ...
"very" note: lean = none, flipper = no, amplifier = YES, ...
So make each note 32 numbers wide.
Call that width EMBEDDING_DIM = 32 (a knob, a number we set).
The notepad is now:
10,000 word-rows x 32 numbers each = 320,000 dials
The word-number just says WHICH ROW to read.
Word 4 means go to row 4, then copy its 32 numbers.
Two things people get wrong about the notepad:
(1) The note slots are NOT labelled by a human.
Nobody writes "slot 3 = flipper."
The notepad starts as 320,000 random junk numbers.
The machine FILLS it itself, by turning those dials a little at a time.
It is nudged by wrongness over many study reviews.
Over many reviews "boring" and "dull" drift to similar notes, and "masterpiece" sits far
away.
So the notepad is WORKSPACE the machine shapes -- it is NOT read off the data.
(2) The notepad is dials, so it counts: 320,000 of them.
That single floor already holds more dials than the rest of the factory combined, as we
will see.
One review through floor 1:
[4, 73, 1, 87, 0, ...] -> look up each number's row -> stack 100 notes
-> a SHEET 100 rows tall x 32 wide
A picture (showing 4 of the 100 rows, 5 of the 32 columns; each note row is one word's
embedding):
word-number note (32 numbers, only 5 shown)
--------- --------------------------------
4 boring [ -0.8 0.1 0.0 0.2 -0.3 ... ]
73 master [ 0.9 -0.1 0.4 0.0 0.7 ... ]
1 the [ 0.0 0.0 -0.1 0.0 0.1 ... ]
87 film [ 0.2 0.3 0.0 -0.2 0.0 ... ]
0 EMPTY [ 0.0 0.0 0.0 0.0 0.0 ... ] (slot 0 stays all-zero)
That stacked sheet -- 100 rows of 32 -- is what floor 2 reads.
Notepad width 4 (not 32). Three words kept. "good" = row 2 with note
[0.9, 0.0, 0.1, -0.2]. A review numbers to [2, 0]. What sheet does floor 1 produce,
and how tall x wide is it if MAX_LEN = 3?
check your slate: MAX_LEN 3 means pad [2, 0] to [2, 0, 0]. Look up each:
row 2 -> [0.9, 0.0, 0.1, -0.2]
row 0 -> [0.0, 0.0, 0.0, 0.0] (empty)
row 0 -> [0.0, 0.0, 0.0, 0.0] (empty)
The sheet is 3 rows tall x 4 wide.
FLOOR 2: THE WALKING WORKER (THE RNN CELL, BY HAND)
IN HAND: floor 1 turned the review into a sheet of 100 notes, each note 32 numbers wide.
Floor 2 must boil that whole sheet down to ONE summary of the review.
This is the heart of the chapter, so we walk it by hand, one word at a time.
This walking worker is what the textbook calls an RNN -- a recurrent neural network, where
"recurrent" means the same dials come round again at every word.
WHY A MEMORY AT ALL.
"not boring" and "boring not" are made of the SAME two words, yet they mean opposite things.
So ORDER carries meaning.
To let "not" flip "boring", the machine must still REMEMBER it saw "not" when it reaches
"boring":
read "not" -> memory: "a flip is pending"
read "boring" -> boring leans down, BUT a flip is pending -> push the verdict UP
So floor 2 keeps a MEMORY.
The memory is a small row of 32 numbers that the worker rewrites after every word.
Think of two pockets on the worker's apron:
pocket A = the MEMORY (32 numbers; before the first word it is ALL ZEROS)
pocket B = the DIALS (frozen during one read-through; only change between
training loops)
The dials in pocket B are two papers and a nudge, REUSED at every single word:
word-dials = a 32 x 32 paper (for the current word's 32 note-numbers)
memory-dials = a 32 x 32 paper (for the old memory's 32 numbers)
nudge = a row of 32 numbers
The word-dials have nothing to do with any particular word.
They are general, the same for "boring" as for "the".
Now the walk.
WORD 1 -- "nolan" (first word, so the old memory is all zeros):
1. see "nolan" -> read its row in the notepad -> pull its 32 numbers
2. WORD-part: nolan's 32 x word-dials (32x32) -> 32 numbers
(row 1 of the paper, times nolan's 32, added up -> new number 1;
row 2 times the same 32 -> number 2; ... 32 rows -> 32 numbers)
3. MEMORY-part: old memory (all zeros) x memory-dials -> all zeros
(anything times zero is zero -- the first word has nothing behind it)
4. ADD them: word-part (32) + memory-part (zeros) -> 32 numbers
5. + NUDGE: add the nudge (32 numbers) -> 32 numbers
6. SQUASH: push each through tanh (crush to between -1 and +1) -> 32 numbers
That result is the NEW memory.
Put it in pocket A, replacing the zeros.
(tanh is a SQUASH: a fixed bend that crushes any single number into the band -1 to +1.
A big positive becomes near +1; a big negative becomes near -1; zero stays 0.
It has no dials and learns nothing -- one crush, at the end.)
WORD 2 -- "ended" (now the old memory is NOT zeros):
1. see "ended" -> notepad -> its 32 numbers
2. WORD-part: ended's 32 x the SAME word-dials -> 32
3. MEMORY-part: pocket A (nolan's leftover memory) x the SAME memory-dials -> 32
(this time NOT zeros, so the past actually contributes)
4. add + nudge + tanh -> the newer memory
5. put it back in pocket A
Same dials as word 1 -- reused.
That reuse is one dial-set looping back over word after word.
That reuse is the whole idea, and it is why the textbook calls this RECURRENT -- the same
dials come round again at every word.
The one recipe to memorise:
new memory = tanh( word's 32 x word-dials + old memory x memory-dials + nudge )
\__ this word folded in __/ \__ the past carried forward _/
Walk all 100 words this way.
After the last word, pocket A holds a 32-number SUMMARY of the whole review, with order
baked in.
ONE WORKER, NOT ONE-PER-WORD. The single most common wrong picture:
NOT 100 workers, one per word
YES 1 worker, who walks word -> word -> word 100 times, SAME dials every word
This is the same reuse trick a photo-scanner uses.
A photo-scanner slides ONE small grid of dials to every spot on the image, reusing it.
Here one worker slides his ONE dial-set across all 100 words, reusing it.
Because the dials are reused, a 100-word review and a 10-word review need the SAME dials.
One worker, one dial-set, just more steps for the longer review.
How the walking worker differs from a photo-scanner that reuses one grid:
photo-scanner: each spot is independent -- no memory between spots
worker (RNN): carries memory FORWARD -- word 2 uses word 1's leftover memory
The dial count for this floor, by hand:
word-dials 32 x 32 = 1,024
memory-dials 32 x 32 = 1,024
nudge 32 = 32
total = 2,080
(Keras reports a SimpleRNN(32) sitting on a 32-wide note as exactly 2,080 -- it counts
(32 + 32) x 32 + 32. Same arithmetic, same answer.)
Memory width 2 (not 32). Old memory = [0, 0] (first word). The word's 2 numbers are
[1, 2]. word-dials = [[1, 0], [0, 1]] (the do-nothing paper), memory-dials anything,
nudge = [0, 0]. tanh(1) ~ 0.76, tanh(2) ~ 0.96. What is the new memory?
check your slate:
WORD-part: row1 . [1,2] = 1x1 + 0x2 = 1 ; row2 . [1,2] = 0x1 + 1x2 = 2 -> [1, 2]
MEMORY-part: old memory is [0,0], so anything x it = [0, 0]
add + nudge: [1, 2] + [0, 0] + [0, 0] = [1, 2]
squash: [tanh(1), tanh(2)] ~ [0.76, 0.96]. New memory ~ [0.76, 0.96].
WHY THE LOOP NEEDS THE TANH SQUASH
IN HAND: the worker rewrites its 32-number memory once per word, 100 times, each time
ending with tanh -- a fixed bend that crushes a number into the band -1 to +1.
Why end with a squash at all?
Here is the problem, made visible.
A squash is a FIXED curve that bends ONE number into a tidy range, alone.
It has no dials, learns nothing, and never looks at neighbours.
Three squashes show up in this kind of machine:
relu (plain hidden clerks): negative -> 0, keep positive range 0 to infinity
tanh (this worker's memory): crush range -1 to +1
sigmoid (the final clerk): crush range 0 to 1
tanh by the numbers: 0 -> 0 ; 1 -> 0.76 ; 2 -> 0.96 ; 5 -> ~1.0 ; -5 -> ~-1.0
Now the problem.
Take the memory and multiply it by the memory-dials 100 times in a row, with NO squash
between.
Suppose one slot's dial run multiplies by 1.5 each word:
1.5 x 1.5 x ... (100 times) = 1.5^100 = a number with 17 zeros -- it EXPLODES.
And if the dial run multiplies by 0.5 each word instead:
0.5^100 = essentially zero -- the memory DIES.
So a 100-step loop with no crusher either blows up to nonsense or collapses to nothing.
The tanh squash pins every memory number between -1 and +1 at EVERY word.
So after 100 words the memory is still a tidy bounded row.
The squash is what keeps the long loop polite -- without it the worker is unusable.
ONE THING THE SQUASH IS NOT: a re-centring trick that peeks at the crowd.
Some machines use a "humbler" that re-centres numbers by PEEKING at a whole handful of
examples at once; it has learned dials (a stretch and a shift) and keeps a running diary.
The tanh squash does none of that.
It peeks at no one, learns nothing, keeps no diary.
It bends each number through the same fixed curve, alone, every time.
FLOOR 3: THE FINAL CLERK (DENSE + S-CURVE)
IN HAND: floor 2 walked all 100 words and left a 32-number summary in pocket A.
One clerk reads those 32 numbers, multiplies each by its dial, adds them and a nudge into
one running total, then squashes that total through the S-curve into a chance between 0
and 1 (the same S-curve derived from odds in Chapter 7):
big positive total -> near 1 (liked)
zero total -> 0.5 (on the fence)
big negative total -> near 0 (did not like)
Dials here: 32 (one per memory-number) + 1 nudge = 33.
The whole factory's dial count, recomputed:
floor 1 notepad 10,000 x 32 = 320,000
floor 2 walking worker (above) = 2,080
floor 3 final clerk 32 + 1 = 33
total = 322,113
Just as in Chapter 9's factory, one floor hogs the dials -- there it was the Dense floor,
here it is the notepad (320,000 of 322,113, about 99%). The famous "recurrent" worker is
a rounding error in the dial budget. The cost is not in the dials; it is in the WALK --
100 steps per review, in order, one after another, which is the bottleneck Part 2's
cousins inherit and the Transformer (a later chapter) finally breaks.
A few places this lab actually tripped people, each one a belief that feels right.
You see 10,000 and 50,000 floating around and take one for the other. They count
different things:
reviews (ROWS) ~ 50,000 how many movie reviews in the pile
distinct words KEPT = 10,000 the vocabulary, a knob you set
Two separate counts that both happen to land in the tens of thousands. 10,000 is
words, never reviews.
Next: boring is word 4 and masterpiece is word 73, so masterpiece must count
heavier? No -- the number is a name tag, not a size:
boring = 4 masterpiece = 73
73 is NOT "more" than 4 -- reshuffle the tags and the words mean the same
That is exactly why floor 1 throws the bare tag away and looks up a learned note
instead.
And the notepad is not read off the reviews like a dictionary. It starts as random
junk and the machine fills it by turning dials:
start: boring -> [ 0.4, -0.1, 0.9, ...] (pure noise)
after training: boring -> [-0.8, 0.1, ...] sits near "dull", far from "great"
It is workspace the machine shapes, the same as any other dial -- not a table
looked up from the data.
The heaviest one: floor 2 is NOT a hundred workers, one per word.
word1 word2 word3 word1 -> word2 -> word3
| | | vs \_____|_____/
w1 w2 w3 one worker walks it
(100 workers?) (same dials every step)
There is one worker. He walks word to word carrying the SAME dial-set at every
step -- which is why a long review and a short one need the same dials. That reuse
is what "recurrent" means.
Then: why carry a memory at all -- why not just average the word-notes? Because
averaging throws ORDER away, and order is where the meaning lives:
"not boring" -> avg(not, boring) --.
>-- identical average, opposite meaning
"boring not" -> avg(boring, not) --'
Only a memory still holding "not" when it reaches "boring" can tell the compliment
from the insult.
Padding with zeros looks like it feeds the machine fake words to learn from. It
does not -- slot 0 is reserved as EMPTY and its notepad row is held all-zero:
[4, 17, 0, 0, 0]
^^^^^^^ look up row 0 -> [0, 0, 0, ...] -> contributes nothing
The padding only makes every review the same length so the fixed dials can eat
them; the zero rows add no signal.
Last: if one number can say "leans positive", why not stop at width 1? Because one
number cannot hold two independent jobs at once:
width 1: not -> [ ? ] say "I flip the next word" AND "no lean of my own"? no
width 32: not -> [lean 0, flips? YES, amplifies? no, ...] room for all of it
Width 32 lets a single word be several things at the same time. The width is a knob.
pile of words, 1/0 per row sentiment classification (binary)
number each word by frequency tokenisation by frequency rank
10,000 kept words vocabulary size (VOCAB_SIZE / num_words)
shared rare-word bin [unknown] / out-of-vocabulary (OOV) token
same-length fix padding / truncating (pad_sequences)
empty slot = 0 pad token (id 0), mask
MAX_LEN = 100 sequence length
notepad embedding layer
one note (32 numbers) word vector / embedding vector
note width 32 embedding dimension (EMBEDDING_DIM)
walking worker, one memory SimpleRNN (recurrent layer)
pocket A (memory) hidden state h_t
word-dials / memory-dials input weights W / recurrent weights U
reuse same dials each word weight sharing across time steps
crush to -1..+1 tanh activation
final clerk + S-curve Dense(1) + sigmoid
wrongness for yes/no binary cross-entropy
downhill-roller Adam optimiser
Nothing above needed a computer; this section is for the day you meet one.
Open the pile and check the balance (Q1):
df = load_data(DATA_PATH) # CSV with columns 'text' and 'label'
q1_shape = df.shape # (how many reviews, 2)
q1_dist = df['label'].value_counts().sort_index() # how many 0s, how many 1s
Number the words, cap the vocabulary, pad to one length, split (Q2):
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from sklearn.model_selection import train_test_split
VOCAB_SIZE = 10000 # keep the 10,000 most common words (a knob)
MAX_LEN = 100 # force every review to 100 numbers (a knob)
tok = Tokenizer(num_words=VOCAB_SIZE, oov_token="<unk>")
tok.fit_on_texts(df['text']) # count words, rank by frequency
seqs = tok.texts_to_sequences(df['text']) # words -> name-tag numbers
X = pad_sequences(seqs, maxlen=MAX_LEN,
padding='post', truncating='post') # 0 = empty slot
y = df['label'].values
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=RANDOM_STATE)
Build the three-floor text-factory (Q3):
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input, Embedding, SimpleRNN, Dense
EMBEDDING_DIM = 32
The shapes flowing through the three floors, one review of 100 word-numbers in:
[100] one review: 100 word-numbers (0 = pad)
| floor 1 (notepad): swap each number for its 32-number note
[100 x 32] a sheet -- 100 word-rows, 32 meaning-numbers each
| floor 2 (walking worker): boil the whole sheet down to one memory
[32] the final 32-number memory (the whole review, summarised)
| floor 3 (one clerk + S-curve)
[1] one chance: liked (near 1) or not (near 0)
model_rnn = Sequential()
model_rnn.add(Input(shape=(MAX_LEN,))) # 100 word-numbers in
model_rnn.add(Embedding(input_dim=VOCAB_SIZE,
output_dim=EMBEDDING_DIM)) # floor 1: the notepad
model_rnn.add(SimpleRNN(32)) # floor 2: walking worker
model_rnn.add(Dense(1, activation="sigmoid")) # floor 3: final yes/no
model_rnn.compile(optimizer="adam",
loss="binary_crossentropy", metrics=["accuracy"])
- 'binary_crossentropy' = wrongness for a yes/no guesser: -log(the chance given to the
true answer). Truth 1, said 0.9 -> tiny; said 0.1 -> huge. (Chapter 7 derives it.)
- 'adam' = the downhill-roller from Chapter 7, Part 2.
- 'model_rnn.summary()' prints 322,113 total dials -- 320,000 in the notepad alone.
Training this factory, watching it forget far-back words, and the two-memory fix that
rescues it (the LSTM) are Part 2.
--> Continue: Chapter 10, Part 2: The Two-Memory Worker
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 10 -- Machines That Read Words):
Part 1 (this post) .
Part 2 -- The Two-Memory Worker .
Part 3 -- The Look-Across Machine
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================