==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
SPECIAL . FOR HACKER NEWS
Attention and the Transformer by Pencil: Every Word Looks at Every Word
============================================================================================
The companion post, The Walking Machine and the Vault, showed why
a signal from word 1 cannot survive eighty rewrites in an RNN, and how the LSTM's uncrushed
vault helps. This post is the harder answer: drop walking entirely. Instead of carrying a
signal through eighty rewrites, lay ALL words out at once and let each word look DIRECTLY at
every other word -- no carrying, no fade. That is attention. The Transformer is what you get
when you stack this on top of a plain classifier.
Two sentences, one word apart at the very end:
"The animal didn't cross the street because IT was too tired."
|________________________________________^ it = the animal
"The animal didn't cross the street because IT was too wide."
|___________________________^ it = the street
Same three letters, "it" -- yet it means the animal in the first line and the
street in the second, and the only clue is a word near the end (tired vs wide). A
machine that reads one word at a time and freezes each into a fixed code is stuck:
its "it" is the identical vector both times, so it cannot tell them apart.
So let every word look at all the others before it settles its meaning. "It"
scores every word in the sentence and blends them in by those scores:
sentence 1: it -> {animal 0.7, street 0.1, tired 0.2} -> "it" leans ANIMAL
sentence 2: it -> {animal 0.1, street 0.7, wide 0.2} -> "it" leans STREET
The same word now carries a different meaning, pulled from whatever surrounds it.
That looking-around and blending is self-attention.
Every attention tutorial either waves its hands ("each word attends to relevant words!") or
buries you in matrix equations. Neither tells you the three things most worth knowing:
(1) why each word needs THREE sticks (want, have, give) rather than one;
(2) where exactly the dot product, the square root, and the softmax come from and what each
does; (3) how 100 words each with their own attention output collapse into a single verdict.
This page shows all three, digit by digit, with one word: "nolan."
TOY WORLD, REAL RECIPE. Pile of movie reviews, each ticked liked (1) or not (0). Our
worked review: "nolan ended" -- two words. Stick width 4 (real: 32). The recipe is exact;
the numbers are the toy.
PROBLEM, DRAWN:
PILE OF REVIEWS (each hand-labelled) MACHINE VERDICT
"the nolan film ended beautifully" -----------> [Transformer] -> 1 (liked)
"boring and a total waste of time" -----------> [Transformer] -> 0 (hated)
"not good at all , skip it" -----------> [Transformer] -> ???
The machine reads ALL words at once -- no walking, no memory carried forward.
To do that, every word must first become a number. Then a stick. Then all sticks look
at each other simultaneously. Only then does the verdict come out.
WORDS ARE NOT NUMBERS -- THE FRONT END (SAME AS THE WALKING MACHINE)
A machine multiplies numbers; it cannot multiply letters. So words become numbers first.
MOVE 1: count every word across the pile, rank by frequency, hand each a number.
DICTIONARY (10,000 most-common words):
the=1 movie=2 was=3 ... nolan=73 ... ended=88 ...
10,000 = VOCAB_SIZE = how many distinct words are kept, not how many reviews,
not how many words per review.
MOVE 2: force every review to 100 slots. Short: fill the end with zeros. Long: chop.
"nolan ended" -> slot-number row:
+----+----+---+---+-----+---+
| 73 | 88 | 0 | 0 | ... | 0 |
+----+----+---+---+-----+---+
^ ^ ^ ^
nolan end padding... padding (100 slots total, 98 zeros)
MOVE 3: swap each slot-number for a STICK -- a fixed-width list of numbers looked up
from a table. The table holds one stick per kept word. Width 4 here (real: 32).
EMBEDDING TABLE LOOKUP:
SLOT NUMBER TABLE ROW STICK (4 wide)
73 -> row 73 -> [ 2, 1, 1, 0 ] <- nolan's stick
88 -> row 88 -> [ 0, 1, 2, 1 ] <- ended's stick
0 -> row 0 -> [ 0, 0, 0, 0 ] <- padding
Stick-numbers start as junk; training tunes them. The review is now a SHEET:
THE SHEET for "nolan ended" (100 rows, each 4 wide):
row 1: [ 2, 1, 1, 0 ] <- nolan
row 2: [ 0, 1, 2, 1 ] <- ended
row 3: [ 0, 0, 0, 0 ] <- padding
...
row 100: [ 0, 0, 0, 0 ] <- padding
This sheet is what the attention step reads. ALL rows at the same time.
WHY THE WALKING MACHINE FADES -- AND WHY WE DROP IT
In the walking machine (the RNN), "nolan" reaches "ended" only through a memory that is
tanh-crushed and rewritten at every word. In an 80-word review, a signal from word 1 must
survive 79 such rewrites. It does not -- it fades to nearly zero before reaching word 80.
The Transformer does not walk. It lays all words out at once and lets every word look
STRAIGHT at every other word, near or far, same cost.
WALK (RNN) -- nolan must travel through every word to reach ended:
nolan -> [rewrite] -> [rewrite] -> ... 79 rewrites ... -> ended
signal from nolan fades with each rewrite. Nearly zero at word 80.
LOOK-ACROSS (Transformer) -- nolan looks directly at ended:
nolan <=====================================> ended (direct, zero rewrites)
nolan <=====================================> word 3 (same cost)
nolan <=====================================> word 80 (same cost)
ALL pairs look at each other simultaneously. No fade. No carrying.
The price of "no walking" is paid elsewhere: the machine must be told word ORDER
explicitly. More on that at the end.
WHY THREE STICKS -- WANT, HAVE, GIVE
Each word arrives as one stick (its embedding). Attention turns it into THREE sticks:
EACH WORD GETS THREE ROLES:
nolan's embedding: [2, 1, 1, 0]
|
+---> WANT-grid --> nolan.WANT = [2, 0, 1, 0] "what am I looking for?"
|
+---> HAVE-grid --> nolan.HAVE = [1, 0, 0, 0] "what do I offer to lookers?"
|
+---> GIVE-grid --> nolan.GIVE = [2, 0, 0, 1] "what I hand over if picked"
Same three grids are reused on EVERY word. One set of grids serves all 100 words.
Why three? One stick cannot serve all three roles at once. A word might be a strong target
(high HAVE) without being a good source of content (low GIVE). Separating the three jobs
into three sticks lets each be tuned independently by training.
HOW A STICK IS MADE FROM A GRID -- a GRID (matrix) times the embedding:
A grid = a table of numbers, one row per output number.
One row "times" the stick: multiply each grid-number against the matching
stick-number, add all products -> ONE number. Repeat for every row -> a NEW stick.
WORKED EXAMPLE -- nolan.WANT from the WANT-grid:
WANT-grid (4 rows x 4 cols) x nolan.embed [2,1,1,0] = nolan.WANT
row 1: [1, 0, 0, 0] -> 1*2 + 0*1 + 0*1 + 0*0 = 2
row 2: [0, 0, 0, 0] -> 0
row 3: [0, 0, 1, 0] -> 0*2 + 0*1 + 1*1 + 0*0 = 1
row 4: [0, 0, 0, 0] -> 0
nolan.WANT = [2, 0, 1, 0]
The HAVE and GIVE sticks are produced the same way, each with their own grid.
MATCH -- NOLAN'S WANT DOTS EVERY WORD'S HAVE
To score how much nolan cares about a word, take nolan's WANT stick and that word's HAVE
stick and DOT them.
DOT PRODUCT: multiply matching numbers pairwise, add all products -> ONE match score.
MATCH SCORES for nolan (nolan.WANT = [2, 0, 1, 0]):
nolan.WANT . nolan.HAVE: nolan.HAVE = [1, 0, 0, 0]
2*1 + 0*0 + 1*0 + 0*0 = 2 <- nolan matches itself a little
nolan.WANT . ended.HAVE: ended.HAVE = [3, 0, 2, 0]
2*3 + 0*0 + 1*2 + 0*0 = 6+2 = 8 <- nolan cares much more about ended
nolan's match row: [ nolan->nolan = 2 , nolan->ended = 8 ]
One score per word. Higher = nolan cares more about that word.
(Every word dots against EVERY other word, including itself -- n words -> n scores.)
SCALE -- DIVIDE BY THE SQUARE ROOT OF THE STICK WIDTH
Wider sticks produce larger dot sums (more products added). The divisor keeps scores
from growing just because the sticks are wide.
stick width = 4
divisor = sqrt(4) = 2
BEFORE SCALING: [2, 8]
÷ 2 ÷ 2
AFTER SCALING: [1, 4]
The divisor is sqrt(stick width) -- a fixed number. Not a data statistic.
(Real stick width = 32; divisor = sqrt(32) ~= 5.66.)
SOFTMAX -- TURN SCORES INTO FRACTIONS THAT ADD TO 1
We want "how much to listen to each word" as fractions -- so they can weight the GIVE
sticks by exactly how much to take from each.
SOFTMAX RECIPE (built from scratch, nothing recalled):
blow each score up by e (e ~= 2.718, the growth number),
then divide each by the TOTAL of all blown-up scores.
APPLIED TO [1, 4]:
SCORES BLOW UP BY e DIVIDE BY TOTAL
------ ---------------- -------------------
1 --> e^1 = 2.718
4 --> e^4 = 54.60
------
total = 57.32
fracs: 2.718 / 57.32 = 0.047 <- listen 4.7% to nolan (self)
54.60 / 57.32 = 0.953 <- listen 95.3% to ended
check: 0.047 + 0.953 = 1.000 <- must always add to exactly 1
nolan listens 0.953 to "ended" and 0.047 to itself. The exponential makes the bigger
score win decisively: 4 vs 1 -> 0.953 vs 0.047, not 0.8 vs 0.2. That sharpening is
what softmax does that plain division would not.
WEIGHTED SUM -- BUILD NOLAN'S NEW STICK FROM THE GIVE STICKS
Use the fractions to mix the GIVE sticks:
FRACTIONS: 0.047 0.953
GIVE STICKS: nolan.GIVE ended.GIVE
[2, 0, 0, 1] [0, 3, 1, 0]
WEIGHTED SUM (multiply each GIVE stick by its fraction, then add number-by-number):
0.047 * [2, 0, 0, 1] = [0.094, 0, 0, 0.047]
0.953 * [0, 3, 1, 0] = [0, 2.859, 0.953, 0 ]
--------------------------------
nolan_new = [0.094, 2.859, 0.953, 0.047]
nolan's new stick is MOSTLY "ended"'s give (0.953 weight). It listened to ended.
SIZE CHECK -- so you are not confused:
We had 2 fractions (one per word). nolan ends with a 4-wide stick, NOT a 2-number list.
The 2 fractions WEIGHT 2 give-sticks (each 4 wide); adding the weighted sticks
number-by-number keeps the width 4. The "how many words" dimension collapses by adding;
the stick width survives. At real scale: 100 fractions weight 100 give-sticks of width 32
-> add -> one 32-wide stick per word.
Every word does this same computation -- look across all words, score, soften, take --
ALL AT ONCE. The sheet goes in; a new sheet of the same shape comes out, each row now
enriched with information from every other row.
TWO TEAMS -- MULTI-HEAD ATTENTION
One set of three grids (want/have/give) catches one kind of link. Run TWO sets side by
side:
MULTI-HEAD ATTENTION (2 heads):
THE SHEET (100 rows x 4 wide)
|
+---------> team 1 (own WANT/HAVE/GIVE grids)
| -> look-across on all 100 words
| -> enriched sheet A (100 rows x 4 wide)
|
+---------> team 2 (different WANT/HAVE/GIVE grids)
-> look-across on all 100 words
-> enriched sheet B (100 rows x 4 wide)
GLUE A and B side by side:
-> enriched sheet (100 rows x 8 wide)
Two teams = "2 heads." The lab uses num_heads=2, key_dim=32 -- two teams, each producing
sticks of width 32. "key_dim" is the width of the WANT/HAVE/GIVE sticks per team.
POSITION -- THE PRICE OF NOT WALKING
Because every word is looked at all-at-once, the raw attention step cannot tell "nolan
ended" from "ended nolan" -- order is invisible when you look everywhere at once.
POSITION MARK: a fixed pattern of numbers encoding slot 1, slot 2, etc. is ADDED
to each word's stick BEFORE the attention step.
slot 1 (nolan): [2,1,1,0] + [pos_1] -> [2.84, 1.54, 1.00, 1.00] (example)
slot 2 (ended): [0,1,2,1] + [pos_2] -> [0.91, 1.84, 2.00, 0.99] (example)
The position mark uses sine and cosine waves at different frequencies.
This is POSITIONAL ENCODING.
Our lab uses a simplified version that skips the explicit position mark -- flagged here
so you know the full machine has it.
COLLAPSING 100 STICKS TO ONE SUMMARY
After attention, there are 100 new sticks (one per word-slot). The verdict needs ONE.
COLLAPSE: average all 100 word-sticks, number by number -> one summary stick.
enriched sheet (100 rows x 4 wide):
row 1: [0.094, 2.859, 0.953, 0.047] <- nolan, after attending
row 2: [0.2, 1.0, 0.5, 0.4 ] <- ended, after attending
rows 3-100: [0, 0, 0, 0] (padding rows, mostly silent)
...
AVERAGE each column:
col 1: (0.094 + 0.2 + 0 + ... + 0) / 100 = 0.003
col 2: (2.859 + 1.0 + 0 + ... + 0) / 100 = 0.039
...
summary stick = [0.003, 0.039, ...] (one 4-wide stick)
The walking machine took its LAST memory as the summary. Here we average all words.
Same end goal: one stick for the verdict.
VERDICT -- PLAIN WORKERS AT THE END
Feed the summary stick through plain workers to a single tick:
VERDICT PIPELINE:
summary stick [4 wide]
|
v
Dropout(0.1) <- zero a random 1-in-10 of the numbers (practice only)
|
v
Dense(20, relu) <- 20 workers. Each: multiply-add + nudge, keep positives.
|
v
Dropout(0.1) <- zero a random 1-in-10 again (practice only)
|
v
Dense(1, sigmoid) <- 1 worker: multiply-add + nudge, crush to 0..1
|
v
0.9 = 90% chance liked. 0.1 = probably not.
relu = keep positives, zero negatives.
sigmoid = crush any number to 0..1 (a probability).
Dropout (zero-a-random-tenth) = forced independence, only during practice, off at exam.
DECODER (JARGON -> PLAIN, MASTER SHEET)
PEACOCK WORD WHAT IT ACTUALLY MEANS
-------------------------- -------------------------------------------------------
embedding a word's stick -- a fixed-width list of numbers
query (Q) a word's WANT stick -- it does the looking
key (K) a word's HAVE stick -- it gets looked at
value (V) a word's GIVE stick -- it gets taken by the winner
dot product multiply pairs, add all -> one number (a match score)
scaled dot-product attention match, divide by sqrt(width), softmax, weighted-sum
softmax blow up by e, divide by total -> fractions adding to 1
attention weight one fraction from softmax (how much to listen)
multi-head attention run two or more want/have/give sets, glue the results
self-attention (Q, K, V) all come from the same sentence (x, x, x)
positional encoding a position-mark added to each word's stick before attention
GlobalAveragePooling average all word-sticks -> one summary stick
relu keep positives, zero negatives
sigmoid crush any number to 0..1 (a probability)
dropout zero a random fraction of numbers (practice only)
dense layer workers: multiply-add + nudge -> one number each
PYTHON: ATTENTION BY HAND, THEN THE FULL MACHINE
Three blocks. The first traces the exact attention math for "nolan" with numpy -- every
number equals the pencil walk above. The second shows the same as one Keras call. The
third is the full transformer model, each line mapped back to the pencil.
import numpy as np
# -----------------------------------------------------------------------
# THE PAINTED FACTS
# review "nolan ended"; stick width 4; made-up tags (same as pencil above)
# -----------------------------------------------------------------------
VOCAB_SIZE = 10000 # kept words
MAX_LEN = 100 # slots per review
EMBEDDING_DIM = 32 # real width (32); pencil uses 4
# =======================================================================
# CASE 1: ATTENTION BY HAND -- numpy, width 4.
# Every number here equals the pencil walk above.
# =======================================================================
print("=== CASE 1: attention for 'nolan' by hand (width 4) ===")
# the two word sticks (from the embedding table)
nolan_embed = np.array([2, 1, 1, 0], dtype=float)
# WANT-grid (4x4); chosen so nolan.WANT = [2,0,1,0]
WANT_grid = np.array([[1,0,0,0],[0,0,0,0],[0,0,1,0],[0,0,0,0]], dtype=float)
nolan_want = WANT_grid @ nolan_embed # [2, 0, 1, 0]
# HAVE and GIVE sticks are hardcoded made-up illustration values,
# exactly as in the pencil paper above. The HAVE-grid would produce them
# at full scale; here we write them directly to keep the arithmetic visible.
nolan_have = np.array([1, 0, 0, 0], dtype=float) # made-up
ended_have = np.array([3, 0, 2, 0], dtype=float) # made-up
nolan_give = np.array([2, 0, 0, 1], dtype=float) # made-up
ended_give = np.array([0, 3, 1, 0], dtype=float) # made-up
print(f"nolan.WANT = {nolan_want}")
print(f"nolan.HAVE = {nolan_have}")
print(f"ended.HAVE = {ended_have}")
# MATCH: nolan.WANT dotted against each word's HAVE
match_nolan = nolan_want @ nolan_have # 2
match_ended = nolan_want @ ended_have # 8
matches = np.array([match_nolan, match_ended])
print(f"\nmatches (dot products): {matches}") # [2, 8]
# SCALE: divide by sqrt(stick width)
stick_width = 4
scaled = matches / np.sqrt(stick_width) # [1, 4]
print(f"scaled (/sqrt({stick_width})): {scaled}")
# SOFTMAX: blow up by e, divide by total -> fractions adding to 1
e_vals = np.exp(scaled) # [2.718, 54.60]
fracs = e_vals / e_vals.sum() # [0.047, 0.953]
print(f"softmax fractions: nolan={fracs[0]:.3f} ended={fracs[1]:.3f} sum={fracs.sum():.3f}")
# WEIGHTED SUM: each GIVE stick times its fraction, add number-by-number
gives = np.vstack([nolan_give, ended_give]) # shape (2, 4)
nolan_new = fracs @ gives # [0.094, 2.859, 0.953, 0.047]
print(f"\nnolan's new stick (after attending): {np.round(nolan_new, 3)}")
print("(mostly ended's give -- it listened 0.953 to ended)")
# nolan.WANT = [2. 0. 1. 0.]
# nolan.HAVE = [1. 0. 0. 0.]
# ended.HAVE = [3. 0. 2. 0.]
# matches: [2. 8.]
# scaled: [1. 4.]
# softmax: nolan=0.047 ended=0.953 sum=1.000
# nolan's new stick: [0.094 2.859 0.953 0.047]
# =======================================================================
# CASE 2: THE SAME ATTENTION AS ONE KERAS CALL.
# MultiHeadAttention(num_heads=2, key_dim=EMBEDDING_DIM)(x, x)
# (x, x) = the sentence looks at itself (self-attention).
# num_heads=2 = run two want/have/give sets, glue the results.
# The math inside is identical to Case 1, done for all 100 words at once.
# =======================================================================
print("\n=== CASE 2: the Keras line that does Case 1 for all 100 words at once ===")
print("# from tensorflow.keras.layers import MultiHeadAttention")
print("# x = MultiHeadAttention(num_heads=2, key_dim=EMBEDDING_DIM)(x, x)")
print("# input shape: (batch, 100, 32) -- 100 words, 32-wide sticks")
print("# output shape: (batch, 100, 32) -- same shape, each word now enriched")
print("# (x, x) = self-attention: want/have/give all come from the same sentence")
# =======================================================================
# CASE 3: THE FULL TRANSFORMER -- every piece mapped to the pencil.
# Build and describe without running training (no IMDB data here).
# =======================================================================
print("\n=== CASE 3: the full transformer (described, not trained) ===")
full_model = """
from tensorflow.keras.layers import (Input, Embedding, MultiHeadAttention,
GlobalAveragePooling1D, Dropout, Dense)
from tensorflow.keras import Model
inputs = Input(shape=(MAX_LEN,))
# [73, 88, 0, ..., 0] -> 100 word-slot numbers
x = Embedding(input_dim=VOCAB_SIZE, output_dim=EMBEDDING_DIM,
input_length=MAX_LEN)(inputs)
# each slot-number -> its 32-wide stick. Output: (batch, 100, 32) sheet.
x = MultiHeadAttention(num_heads=2, key_dim=EMBEDDING_DIM)(x, x)
# every word: make WANT/HAVE/GIVE, dot, scale, softmax, weighted-sum of GIVE.
# 2 teams, results glued. Output: (batch, 100, 32) -- enriched sheet.
x = GlobalAveragePooling1D()(x)
# average all 100 word-sticks -> one 32-wide summary stick. (batch, 32)
x = Dropout(0.1)(x) # zero 1-in-10, practice only
x = Dense(20, activation='relu')(x) # 20 workers, keep positives
x = Dropout(0.1)(x)
outputs = Dense(1, activation='sigmoid')(x) # 1 worker -> 0..1 tick
model = Model(inputs, outputs)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# binary_crossentropy: wrongness = -log(chance given to the true tick)
# said 0.96 for truth=1: -log(0.96) = 0.04 (small)
# said 0.10 for truth=1: -log(0.10) = 2.30 (big)
# adam: the downhill-roller that tunes all dials
history = model.fit(X_train, y_train, epochs=5, batch_size=64,
validation_data=(X_test, y_test))
# 5 passes over the study pile, 64 reviews per handful
"""
print(full_model)
Output of Case 1 (Cases 2 and 3 are descriptions, not run here):
=== CASE 1: attention for 'nolan' by hand (width 4) ===
nolan.WANT = [2. 0. 1. 0.]
nolan.HAVE = [1. 0. 0. 0.]
ended.HAVE = [3. 0. 2. 0.]
matches (dot products): [2. 8.]
scaled (/sqrt(4)): [1. 4.]
softmax fractions: nolan=0.047 ended=0.953 sum=1.000
nolan's new stick (after attending): [0.094 2.859 0.953 0.047]
(mostly ended's give -- it listened 0.953 to ended)
> THE ONE HONEST FOOTNOTE. This page shows self-attention for the ENCODER only -- the
> sentence looking at itself. A full Transformer for translation has TWO types of attention:
> encoder self-attention (source sentence looks at itself), decoder self-attention (target
> sentence looks at itself, with future words masked out), and cross-attention (decoder looks
> at the encoder's output). The classifier here has no decoder and no masking because the job
> is a single tick (liked / not), not a sequence output. The attention math is identical in
> all three types; only what plays the role of WANT vs HAVE differs. Also: positional
> encoding (sine/cosine position marks added before attention) is present in the full
> Transformer but skipped in the lab's simplified version -- named here so it is not smuggled.
---
Companion: The Walking Machine and the Vault shows why the LSTM
exists and what the vault arithmetic looks like by hand -- the history that makes the
Transformer's "no walking" choice meaningful.
The Agent Ladder and Cheapest Walk: UCS and A*
build the search-and-agents corners of the same toolbox.
---
One Breath: words become sticks (embedding table, one stick per kept word), then each word
makes THREE new sticks from its own -- WANT (what it looks for), HAVE (what it offers),
GIVE (what it hands over if picked) -- via the same three grids reused on every word; WANT
dotted against every word's HAVE gives match scores (nolan->ended = 8), divided by sqrt of
stick-width to keep scores from growing with width, then softmax turns them into fractions
adding to 1 (0.047 self, 0.953 ended), and a weighted sum of GIVE sticks builds the new
stick (mostly ended's give); this runs for every word at once, with no walking and no fade;
two teams of grids run in parallel and their outputs are glued (multi-head); the 100
resulting sticks are averaged into one summary stick, passed through two dense layers and a
sigmoid, and the result is the tick.
----------------------------------------------------------------------------------------------
home . archive . source on GitHub
==============================================================================================