==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 6 . FINDING PATTERNS WITHOUT ANSWERS . PART 2 OF 6
The Strongest Direction: Crushing a Many-Wall Room Into a Flat Page
============================================================================================
Say each row is a thing and each column is a number measured on it.
The gap between two rows is the straight-line (Euclidean) distance.
To get it: subtract the two rows column by column.
Then square each difference.
Then add the squares.
Then take the square root.
With 3 columns this is easy.
Three columns means three rulers at right angles, which you can picture.
But what if you have 13 columns?
Or 100?
You cannot draw a 13-wall room on flat paper.
Which means the dots live in a space too many-walled to picture.
PCA (Principal Component Analysis) is the trick that CRUSHES that many-wall room down
to a flat page.
It keeps the shape of the dots while doing so.
The idea is simple.
Shine a flashlight on the cloud of dots from different angles.
The shadow that casts the LONGEST spread is the first "principal component."
The next-longest shadow, at a right angle to the first, is the second.
So crush to those two shadows and draw what you see.
Crushing a 3D object down to a flat picture means killing one direction -- the
only question is which. Look straight down on a tall bottle and its height is gone:
from above from the side
+---+ ___
| | / \
+---+ | | same bottle, two shadows: overhead
a flat blob | | throws the shape away, the side
(height lost) \___/ keeps it
You have to drop a dimension; drop the wrong one and the shadow says nothing. So
do not pick by eye -- spin the cloud and watch its shadow stretch and shrink, then
keep the angle where the shadow is WIDEST:
narrow shadow -> . (points pile on top of each other)
wide shadow -> . . . . . . . (points stay spread, every one distinct)
Widest shadow means the most spread survives the crush, which means the most
truth kept. That widest direction is the first principal component, and PCA
crushes along it.
THE SHEET
alcohol, malic_acid, ash, alcalinity_of_ash, magnesium, total_phenols,
flavanoids, nonflavanoid_phenols, proanthocyanins, color_intensity,
hue, od280/od315_of_diluted_wines, proline
178 wines. 13 chemical measurements. One row per wine.
wine alcohol malic_acid ash ... proline
----- ------- --------- ---- ... -------
1 14.23 1.71 2.43 ... 1065.0
2 13.20 1.78 2.14 ... 1050.0
3 13.16 2.36 2.67 ... 1185.0
... ... ... ... ... ...
Look at the columns.
Alcohol is ~13.
Proline is ~1000.
Ash is ~2.
A column measured in thousands will dominate a column measured in single digits.
This happens because its raw gaps are bigger, not because it matters more.
So the fix is to standardise every column first.
To standardise a value: standardised value = (raw value - column mean) / column spread.
Here column mean is the average of that column, and column spread is its standard deviation (how far values sit from that average on average).
This puts every column on mean 0, spread 1.
Which means no column shouts louder than another.
The result is the same ruler for all 13.
Now count the standardising in clerk-strokes.
178 wines x 13 columns = 2,314 numbers to standardise.
Each number costs one subtract and one divide = 2 strokes.
So 2,314 x 2 = 4,628 strokes before PCA even begins.
Then one PC score per wine costs 13 multiplies + 12 adds = 25 strokes.
So all 178 wines on one component run 178 x 25 = 4,450 strokes.
A room of clerks clears the whole crush by lunch.
You would still be sharpening your pencil.
CORE METAPHOR: FLASHLIGHT AND SHADOWS
Imagine each wine is a dot floating in a 13-wall room.
You cannot draw this room.
But you CAN shine a flashlight through it and trace the shadow on the wall.
| flashlight |
| * | 13-wall room (invisible)
| * * |
| * * * |
| |
----------------
shadow on the wall
The shadow flattens the 13 walls into 1 line.
The dots that were far apart in the room land far apart on the shadow.
The dots that were close land close.
Rotate the flashlight.
The shadow gets longer or shorter.
The LONGEST shadow is the one that spreads the dots out the most.
That longest-shadow direction is the first principal component (PC1).
A principal component is a direction through the cloud; the shadow of the dots onto it is what you keep.
So PC1, the longest shadow, is the single most informative way to look at the data.
Now rotate 90 degrees from PC1.
The next-longest shadow at that right angle is PC2, the second principal component.
Now you have two shadows (PC1 and PC2) at right angles.
Plot each wine at (its PC1 coordinate, its PC2 coordinate).
That dot on flat paper captures MORE of the original structure than any other two-number summary.
A 2-WALL WORKED EXAMPLE (NOT 13, SO YOU CAN SEE IT)
Take 2 measurements (alcohol, color_intensity) for 4 wines.
Two measurements means you can draw the room on paper.
Which means you can see the shadow with your own eyes.
wine alcohol color_intensity
----- ------- ---------------
A 13.0 4.0
B 13.5 6.0
C 14.0 5.0
D 14.5 7.0
First, standardise each column.
Standardise means: standardised value (the z-score) = (raw value - column mean) / column spread.
This puts both columns on mean 0, spread 1, so neither shouts louder.
The column mean is the average of the column; the column spread is its standard deviation.
alcohol mean=13.75, spread=0.65
color mean=5.5, spread=1.29
A: alcohol z = (13.0-13.75)/0.65 = -1.15, color z = (4.0-5.5)/1.29 = -1.16
B: alcohol z = (13.5-13.75)/0.65 = -0.38, color z = (6.0-5.5)/1.29 = 0.39
C: alcohol z = (14.0-13.75)/0.65 = 0.38, color z = (5.0-5.5)/1.29 = -0.39
D: alcohol z = (14.5-13.75)/0.65 = 1.15, color z = (7.0-5.5)/1.29 = 1.16
Notice the two columns mostly rise and fall together: the wine low in alcohol (A) also
runs low in colour, the high one (D) runs high in both. That shared rise is exactly what
gives PCA a clear diagonal to catch -- the longest shadow lies along that diagonal.
Then find the longest shadow.
In a 2-walled room this means spinning a line until the dots spread along it as far as possible.
The answer comes by formula, not by actually spinning a flashlight.
The answer is a weighted combination of the two columns:
PC1 = 0.71 * alcohol_z + 0.71 * color_z
This is a RECIPE.
Take 0.71 parts of the alcohol z-score.
Add 0.71 parts of the color z-score.
The two weights, 0.71 and 0.71, are called the loadings: how much of each column goes into the PC.
The resulting number is that wine's PC1 coordinate, also called its PC1 score.
A: 0.71 * (-1.15) + 0.71 * (-1.16) = -0.82 + -0.82 = -1.64
B: 0.71 * (-0.38) + 0.71 * (0.39) = -0.27 + 0.28 = 0.01
C: 0.71 * (0.38) + 0.71 * (-0.39) = 0.27 + -0.28 = -0.01
D: 0.71 * (1.15) + 0.71 * (1.16) = 0.82 + 0.82 = 1.64
PC1 spreads from -1.64 to +1.64.
PC2 is the next shadow at a right angle to PC1.
It has its own recipe (its own loadings):
PC2 = -0.71 * alcohol_z + 0.71 * color_z
A: -0.71 * (-1.15) + 0.71 * (-1.16) = 0.82 + -0.82 = 0.00
B: -0.71 * (-0.38) + 0.71 * (0.39) = 0.27 + 0.28 = 0.55
C: -0.71 * (0.38) + 0.71 * (-0.39) = -0.27 + -0.28 = -0.55
D: -0.71 * (1.15) + 0.71 * (1.16) = -0.82 + 0.82 = 0.00
Plot each wine at (PC1, PC2):
PC2 ^
0.5 | B
|
0.0 A----------------------------D--> PC1
| C
-0.5 |
+-----+-----+-----+-----+-----+
-1.5 -0.75 0 0.75 1.5
A and D sit far apart along PC1 (at -1.64 and +1.64) but dead on the PC2 line.
B and C barely move along PC1 yet split apart on PC2.
The dots spread far more along PC1 than PC2.
Which means PC1 captures the stronger pattern.
A fifth wine E (made-up) lands at alcohol_z = 1.0 and color_z = 1.0.
Work its PC1 and PC2 from the two recipes above.
PC1 recipe: PC1 = 0.71 * alcohol_z + 0.71 * color_z.
PC2 recipe: PC2 = -0.71 * alcohol_z + 0.71 * color_z.
check your slate: PC1 = 0.71 * 1.0 + 0.71 * 1.0 = 0.71 + 0.71 = 1.42;
PC2 = -0.71 * 1.0 + 0.71 * 1.0 = -0.71 + 0.71 = 0.
E sits far out along PC1 and dead centre on PC2.
So E is a wine the strongest shadow finds extreme.
HOW MUCH DOES EACH SHADOW CAPTURE?
IN HAND so far: four wines put on the same ruler (mean 0, spread 1 per column).
Then a recipe PC1 = 0.71 * alcohol_z + 0.71 * color_z that spread them from -1.64 to +1.64.
Then PC2 = -0.71 * alcohol_z + 0.71 * color_z at a right angle.
The four PC1 scores were -1.64, 0.01, -0.01, 1.64.
The four PC2 scores were 0.00, 0.55, -0.55, 0.00.
This section asks how much of the total spread each shadow actually caught.
First define variance.
Variance is the average of the squared distances from the mean.
It measures how spread out a set of numbers is.
Here the mean of every PC score set is 0, because the data was centred.
So variance = mean of the squared scores.
The kept fraction (also called explained variance ratio) for a PC is:
that PC's variance divided by the total variance of all PCs.
PC1 explained fraction = variance_of_PC1_scores / (variance_of_PC1 + variance_of_PC2)
We already have the PC scores.
Variance = mean of squared values, since the mean is 0:
PC1 scores: -1.64, 0.01, -0.01, 1.64
PC1 variance = (1.64^2 + 0.01^2 + 0.01^2 + 1.64^2) / 4
= (2.69 + 0.00 + 0.00 + 2.69) / 4
= 5.38 / 4 = 1.35
PC2 scores: 0.00, 0.55, -0.55, 0.00
PC2 variance = (0.00^2 + 0.55^2 + 0.55^2 + 0.00^2) / 4
= (0.00 + 0.30 + 0.30 + 0.00) / 4
= 0.60 / 4 = 0.15
total = 1.35 + 0.15 = 1.50
PC1 fraction = 1.35 / 1.50 = 0.90 (90%)
PC2 fraction = 0.15 / 1.50 = 0.10 (10%)
PC1 carries about 90% here, PC2 only 10% -- because the two columns rise and fall
together (they correlate), so most of the spread lies along the one diagonal PC1 catches.
Had the two columns been unrelated, each PC would carry about half instead.
The fractions get even more lopsided when you have 13 columns.
Then the first shadow can eat 36% while the 12th shadow carries 1%.
With the real 13-column wine data, PC1 carries ~36%.
PC2 carries ~19%.
The rest scatters across PCs 3-13.
WHY KEPT + LOST ALWAYS ADDS BACK TO THE SAME TOTAL
Here is the lock that makes the fraction trustworthy.
Take one dot and the line it casts its shadow on.
Draw three lengths:
>> Do this one after reading the three lengths just below: A dot's shadow on the line is 4 long.
Its perp (the walk off the line to the dot) is 3 long.
How long is the stick from the origin to the dot?
check your slate: stick^2 = shadow^2 + perp^2 = 4*4 + 3*3 = 16 + 9 = 25, so
stick = sqrt(25) = 5.
This is the 3-4-5 triangle.
The kept length (shadow) and the lost length (perp) always square back to the same fixed stick.
stick = straight line from the middle (origin) to the dot
shadow = how far the dot's shadow lands along the line (KEPT)
perp = the dot's sideways gap off the line (LOST)
* dot
/|
stick | perp (off the line)
/ |
-+---+-------- the line
middle shadow (along the line)
Those three make a right angle.
So Pythagoras locks them:
stick^2 = shadow^2 + perp^2
The stick was fixed the moment you measured the dot.
Spinning the line never changes it.
Spinning only shuffles the split between shadow and perp.
Now add this over all the dots:
sum of stick^2 = sum of shadow^2 + sum of perp^2
(what the dots ARE) (what the drawing KEEPS) (what flattening LOSES)
The left side never moves.
So the line that KEEPS the most has the biggest sum of shadow^2.
And that same line LOSES the least, the smallest sum of perp^2.
Most-kept and least-lost are the same line seen from two sides.
Which is why PC1, the longest-shadow direction, is also the smallest-reconstruction-error direction.
(Reconstruction is rebuilding the dots from the shadow; its error is the leftover perp.)
And the kept fraction is just:
kept fraction = sum of shadow^2 / sum of stick^2
CHOOSING HOW MANY SHADOWS TO KEEP
Add the fractions from the biggest shadows downward until you hit 80%:
PC1: 36% cumulative: 36%
PC2: 19% cumulative: 55%
PC3: 11% cumulative: 66%
PC4: 7% cumulative: 73%
PC5: 5% cumulative: 78%
PC6: 4% cumulative: 82% <- past 80%
With 5 PCs you have ~78%.
With 6 PCs you have ~82%.
So most of the structure is captured in the first 5 or 6 shadows.
The remaining 7 PCs carry mostly noise.
RECIPE (LOADINGS)
Each PC is a RECIPE.
The recipe says how much of each original column goes into it.
PC1 = 0.14 * alcohol + 0.16 * malic_acid + ... + 0.32 * proline
Those weight numbers are the loadings.
A loading is how much of one column goes into one PC.
The loadings tell you which columns the PC leans on.
A high loading means that column matters a lot for that PC.
For PC1 on the wine data, the highest loading is often on proline or flavanoids.
These columns vary the most, so they drive the longest shadow.
SCORES (TRANSFORM)
Every wine gets a PC1 score and a PC2 score.
A score is the wine's coordinate along one PC, computed by running that PC's recipe.
These two scores are the coordinates you plot.
So the original 13 columns are crushed into 2 numbers, which is 1 dot on a flat page.
wine PC1 PC2
----- ------ ------
1 2.13 -0.45
2 1.62 -0.85
3 2.05 0.32
... ... ...
178 -2.41 0.78
The scatter plot of PC1 vs PC2 shows the cloud of wines crushed to 2D.
Each dot is one wine.
Dots close together mean chemically similar wines.
BLOWING IT BACK UP (RECONSTRUCTION)
Reconstruction means rebuilding the original columns from the kept scores.
To do it: take only PC1 and PC2 scores and multiply each back by its loadings.
This gives a BLURRY version of the original 13 columns.
It is blurry because you threw away PCs 3-13.
original alcohol = 14.23
reconstructed (2 PCs) ~= 13.85 (off by ~0.4)
The more PCs you keep, the less blur.
The blur is measured by MSE, the mean squared error.
MSE is the average of the squared gaps between original and rebuilt values.
keep 2 PCs: MSE ~ 0.45
keep 5 PCs: MSE ~ 0.19
keep all 13: MSE = 0.00 (perfect, but pointless)
Now line those numbers up against the kept fractions from earlier.
The kept fraction is the share of total spread the kept PCs hold.
keep 2 PCs: kept ~ 55% -> lost ~ 45% -> MSE ~ 0.45
keep 5 PCs: kept ~ 80% -> lost ~ 20% -> MSE ~ 0.19
This is not a coincidence.
On standardised data the reconstruction MSE IS the thrown-away fraction.
This is the same kept + lost = fixed total from the stick-shadow-perp section, read from the lost side.
So you never need to run the reconstruction to know its error.
1 minus the kept fraction already told you.
Reconstruction error measures how much structure was lost when you crushed the room.
Your goal is to lose as little as possible.
And still be able to draw the picture on flat paper.
A fair few places this bites, each a thing that feels right.
PCA hunts the direction of greatest SPREAD, so a column with huge numbers grabs it
by default:
proline spread ~ 1000 ash spread ~ 1
PC1 -> basically "proline", everything else ignored
Standardise first, or the first PC is just "the column with the biggest numbers".
A loading is the recipe for the PC, not a correlation with it:
loading 0.5 -> "half a part of this column goes INTO the PC"
NOT -> "this column correlates 0.5 with the PC"
Different numbers entirely -- the loading is a mixing weight.
PCA forces every PC to sit at a RIGHT ANGLE to the others, so their scores come out
uncorrelated:
PC2
| . . .
| . . . cloud shows no tilt -> PC1 and PC2 uncorrelated
+----------- PC1
If your PC1 and PC2 come out correlated, the fit or the data is wrong.
Keep all 13 PCs and reconstruction is perfect (MSE 0) -- but you kept all the noise
with it:
all 13 PCs: MSE 0, noise and all (no compression, no point)
top few PCs: small MSE, noise dropped
The whole point is to drop noisy dimensions. The 80% cumulative line is a rule of
thumb, not a law.
inverse_transform hands the numbers back in the STANDARDISED space, not raw units:
PCA inverse_transform -> mean 0, spread 1 (still standardised)
for real units -> also undo the standardisation: x*spread + mean
Skip the second step and your "reconstructed" wine is in z-scores, not the original
scale.
Two cousins, easy to grab the wrong one:
explained_variance_ = the raw spread value
explained_variance_ratio_ = the FRACTION (0..1) of total spread
Use the ratio for "how much is captured" and for the cumulative plot.
On the PC1-PC2 scatter, the dot farthest from (0, 0) is the most extreme wine in the
crushed view:
PC2
| . <- farthest from origin = most distinctive (or an outlier)
| . .
+----------- PC1
Worth a look -- it is either an outlier or a very distinctive profile.
When PC1+PC2 keep only, say, 40% of the spread, the other 60% died in the sideways
gaps -- and that lost 60% can drop two truly-far dots onto the same spot:
truth (high-D): A .......... B (far apart)
drawing (40%): A B (printed on top of each other)
Two real islands can print as one, or one as two. Trust the islands you see only
when the kept fraction is high; a low fraction means the flat page is hiding most of
the real placement.
Last, the kept fraction is a slice of the spread pie over the whole pie, not a
per-head number:
RIGHT: sum(shadow^2) / sum(stick^2) -> lands in [0, 1]
WRONG: sum(shadow^2) / (number of dots) -> "spread per dot", not a fraction
Divide by the total spread, never the dot count, or it will not sit between 0 and 1.
Plain term used above Standard label
----------------------------------- ------------------------------------------
crush a many-wall room to flat dimensionality reduction
stick from middle to dot the centred data vector (its norm)
shadow along the line (kept) the PC score / projection
sideways gap off the line (lost) the reconstruction residual
stick^2 = shadow^2 + perp^2 Pythagoras / orthogonal decomposition
longest shadow / strongest direction first principal component (PC1)
second shadow (at right angle) second principal component (PC2)
the recipe for a shadow loadings / components_
each wine's coordinate on the shadow score / transformed data
how much each shadow carries explained variance ratio (PVE)
keep shadows until 80% captured cumulative PVE threshold
blow the shadow back up inverse transform / reconstruction
blurriness after blowing up reconstruction error (MSE)
standardise before crushing StandardScaler before PCA
column importance in the recipe loading magnitude (absolute value)
the 13 numbers crunched into 2 2D embedding / projection
Nothing above needed a computer -- only pencils, clerks, and patience. This last
section is for the day you meet one: the same four wines, the same numbers, in Python.
No loop, no hidden helper -- every wine written on its own line, exactly as on the slate,
so you can run it and watch the pencil numbers print back.
---- THE FOUR WINES, BY HAND, ONE LINE EACH (NO LOOP) ----
COLUMN AVERAGE = ADD THE FOUR, DIVIDE BY FOUR
alc_mean = (13.0 + 13.5 + 14.0 + 14.5) / 4 # 13.75
col_mean = (4.0 + 6.0 + 5.0 + 7.0 ) / 4 # 5.5
COLUMN SPREAD = SUBTRACT MEAN, SQUARE, ADD, DIVIDE BY (4-1), TAKE THE ROOT
alc_spread = (((13.0-13.75)**2 + (13.5-13.75)**2 + (14.0-13.75)**2 + (14.5-13.75)**2)/3)**0.5 # ~0.65
col_spread = (((4.0 -5.5 )**2 + (6.0 -5.5 )**2 + (5.0 -5.5 )**2 + (7.0 -5.5 )**2)/3)**0.5 # ~1.29
STANDARDISE EACH WINE: Z = (RAW - MEAN) / SPREAD, ONE LINE PER WINE
A_alc_z = (13.0 - 13.75)/0.65; A_col_z = (4.0 - 5.5)/1.29 # -1.15, -1.16
B_alc_z = (13.5 - 13.75)/0.65; B_col_z = (6.0 - 5.5)/1.29 # -0.38, 0.39
C_alc_z = (14.0 - 13.75)/0.65; C_col_z = (5.0 - 5.5)/1.29 # 0.38, -0.39
D_alc_z = (14.5 - 13.75)/0.65; D_col_z = (7.0 - 5.5)/1.29 # 1.15, 1.16
PC1 SCORE = 0.71*ALC_Z + 0.71*COL_Z, WORKED FOR EACH WINE ON ITS OWN LINE
A_pc1 = 0.71*A_alc_z + 0.71*A_col_z # -1.64
B_pc1 = 0.71*B_alc_z + 0.71*B_col_z # 0.01
C_pc1 = 0.71*C_alc_z + 0.71*C_col_z # -0.01
D_pc1 = 0.71*D_alc_z + 0.71*D_col_z # 1.64
PC2 SCORE = -0.71*ALC_Z + 0.71*COL_Z (THE RIGHT-ANGLE RECIPE)
A_pc2 = -0.71*A_alc_z + 0.71*A_col_z # 0.00 (A and D land on ~0 here)
B_pc2 = -0.71*B_alc_z + 0.71*B_col_z # 0.55
C_pc2 = -0.71*C_alc_z + 0.71*C_col_z # -0.55
D_pc2 = -0.71*D_alc_z + 0.71*D_col_z # 0.00
print("PC1:", round(A_pc1,2), round(B_pc1,2), round(C_pc1,2), round(D_pc1,2))
print("PC2:", round(A_pc2,2), round(B_pc2,2), round(C_pc2,2), round(D_pc2,2))
HOW MUCH SPREAD EACH SHADOW CAUGHT: VARIANCE = MEAN OF SQUARED SCORES (MEAN IS 0)
pc1_var = (A_pc12 + B_pc12 + C_pc12 + D_pc12) / 4 # 1.35
pc2_var = (A_pc22 + B_pc22 + C_pc22 + D_pc22) / 4 # 0.15
print("PC1 fraction:", round(pc1_var/(pc1_var+pc2_var), 2)) # 0.90
print("PC2 fraction:", round(pc2_var/(pc1_var+pc2_var), 2)) # 0.10
(A and D print as a hair off 0.00 on PC2 -- that is rounding dust from writing the
spreads as 0.65 and 1.29 instead of their longer decimals; with full precision they are
exactly 0.)
Now the real sheet -- 178 wines, 13 columns -- is the one place a loop earns its keep.
You cannot hand-type 178 rows, and the longest-shadow direction across 13 columns is not
something a pencil finds; it is the eigenvector of a 13x13 grid. THIS is where the toolbox
is honest abstraction, not hiding: the four-wine block above shows you exactly what each
line below does, just scaled past what a hand can reach.
>> NEW TO PYTHON? Each named once:
StandardScaler() -- the same (raw - mean)/spread, applied to every column
PCA() -- the shadow-finder (eigenvectors of the standardised grid)
.explained_variance_ratio_ -- the kept fraction per PC (our pc1_var/total, automated)
.components_ -- the loadings (the 0.71-style recipe, one row per PC)
.transform(X) -- each wine's PC scores (our A_pc1 ... D_pc1, automated)
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
df = pd.read_csv("wine.csv") # 178 rows, 13 columns
X_scaled = StandardScaler().fit_transform(df) # every column to mean 0, spread 1
pca = PCA().fit(X_scaled)
print("PC1 fraction:", round(pca.explained_variance_ratio_[0], 2)) # ~0.36
print("PC2 fraction:", round(pca.explained_variance_ratio_[1], 2)) # ~0.19
scores = pca.transform(X_scaled) # each wine's 13 PC coordinates
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 6 -- Finding Patterns Without Answers):
Part 1 -- Looking at a Sheet With No Answers .
Part 2 (this post) .
Part 3 -- Grouping by Nearest Centre (K-Means) .
Part 4 -- The Family Tree (Hierarchical Clustering) .
Part 5 -- Both Tools on NCI60 (Re-visited) .
Part 6 -- Filling the Blanks (Recommender Systems)
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================