==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 1 . PREDICTING HOUSE PRICES . PART 1 OF 3
Guessing House Prices, End to End
============================================================================================
Welcome -- this is where the blog begins.
It asks nothing of you but curiosity and a pencil.
No prior knowledge.
No jargon you have not earned.
No hand-waving past the hard parts.
Here is the promise.
By the end of this single post you will have built a machine that guesses the price of a house it has never seen.
You build it by hand, on a real sheet of California house prices.
It is not a toy version.
It is the same machinery the textbooks dress up in Greek letters.
The difference here is that we build it first and name it last.
Two later posts then take the two rules apart screw by screw.
For now, stand back and see the whole thing at once.
THE 1950 CONTRACT
Pretend it is 1950. Print these pages. Pencils, graph paper, a blackboard --
and a room of infinite, tireless clerks for the heavy arithmetic. No computer,
no calculator, anywhere in the teaching. Every number is recomputed where it is
needed (no page assumes you remember the last one), every worked example is
followed by a drill for your slate, and every cost is counted in
clerk-steps. Your attention is the scarce thing; arithmetic is free.
>> HOW THIS BOOK IS LAID OUT (read once, then forget)
This blog is really a short book in six chapters (twenty-one posts), meant to be read top to bottom.
Every post draws the idea by hand first -- plain words, a pencil sketch, simple
arithmetic. Any runnable Python is gathered into a "The Code, If You Want It"
section at the very END of each post, so you can follow the whole story without ever
stepping over code. That code assumes almost no Python: the handful of things you
need are explained in one short line the first time they appear. Skip the code or
study it -- the idea stands on its own either way.
JOB: GUESS FROM A SHEET OF NUMBERS
rooms income age ocean people | PRICE <- right-answer column
--------------------------------------------------------
4.2 3.5 25 1.0 320 | 1.4
5.1 5.6 10 0.5 210 | 2.8
3.8 2.1 42 3.2 890 | 0.9
+--------- 8 measured columns ----+| the answer
The sheet has 20,640 rows.
Each row is one California neighbourhood.
Each row has 8 measured columns -- things like rooms and income.
Each row also has one right-answer column: median house price in $100,000s.
Median = the middle value when all prices in a neighbourhood are sorted low to high.
Five houses at prices 0.8, 1.4, 1.4, 2.1, 3.5 (in $100,000s):
sorted: 0.8 1.4 1.4 2.1 3.5
^
median = 1.4 (the 3rd of 5 -- the one in the exact middle)
Plain average of those same five: (0.8 + 1.4 + 1.4 + 2.1 + 3.5) / 5 = 9.2 / 5 = 1.84.
The median sits where most houses actually sit.
The average gets dragged up by the one expensive outlier (3.5).
So a price of 1.4 means $140,000.
The job sounds simple and hides the whole game.
The job: build a rule that guesses the right answer for a row it has NEVER seen.
That means a row it never studied, not a row it already memorised.
Anyone can memorise answers they have been shown.
The trick is being right about the houses you have not seen.
First you need a way to say HOW wrong a guess is, and the obvious way quietly
lies. Take two houses, both guessed at 100:
House A truth 120, guess 100 miss = +20 (guessed too low)
House B truth 80, guess 100 miss = -20 (guessed too high)
Average those raw misses and the two slide toward each other and meet at zero:
-20 0 +20
|--------------+--------------|
House B House A
\____________ + ____________/
= 0
Zero. The guesser looks flawless -- yet it was off by 20 on both houses. The
plus and the minus ate each other and hid the damage. So raw misses are out.
Square each miss first and the sign has nothing left to cancel with:
(+20)^2 = 400, (-20)^2 = 400, average 400. Both errors now count as size, not
direction. That squaring is the whole reason mean squared error squares first.
ONE RULE: HIDE SOME ROWS
ALL ROWS
################----
working pile (80%) hidden pile (20%)
^ locked away -- read once, at the very end
Before touching anything else, do the one thing every honest guesser does first.
Set aside a random 20% of rows and lock them in a drawer.
Call this locked-away 20% the hidden pile.
Call the other 80% the working pile -- the rows you build the rule from.
The hidden pile's right answers must not touch any decision.
They must not touch the rule.
They must not touch any setting.
They must not even touch how the numbers are prepared.
They exist for a single honest check, taken once, at the very end.
Why be so strict?
The reason is mechanical, not moral.
Any rule checked on the same rows it was built from reports a flattering, false-low mistake.
That is like grading your own exam with the answer key open.
Take the ask-closest rule at k=1 as a witness.
The ask-closest rule guesses by copying the answer of the most similar working-pile row.
Here k=1 means copy the answer of the single most similar row.
On its own working pile this rule makes ZERO mistakes.
That is because every row's most similar row is ITSELF.
So it copies its own true answer every time.
But that rule then stumbles on new rows.
Which means the drawer is the only number that does not lie.
SHUFFLE FIRST
Rows often arrive sorted by district. Shuffle with a fixed starting point so the
split is reproducible and not ordered. The fixed point controls reproducibility; the
shuffle removes order bias.
RULE 1 -- ASK THE CLOSEST ROWS
IN HAND: a sheet of 20,640 rows.
Each row has 8 measured columns plus one right-answer column (price).
Cut the rows 80/20 into a working pile and a hidden pile.
Working pile = 20,640 x 8/10 = 16,512 rows on the desk.
Hidden pile = 20,640 - 16,512 = 4,128 rows locked in the drawer.
Now the first guessing rule.
new row ?
|
measure gap to every working-pile row
|
pick the k closest
|
average their right-answers -> guess
gap(a, b) = sqrt( sum_j (a_j - b_j)^2 )
Read the symbols in that line.
a is the new row; b is a working-pile row.
a_j is the number in column j of the new row; b_j is the number in column j of the working row.
sum_j means add over every column j.
So the gap subtracts the two rows column by column, squares each difference, adds them, and takes the square root.
One honest IOU, named out loud: this formula is handed to you here, not derived.
WHY square-then-root measures distance is its own job for a later post.
The short reason: it is the ruler-on-graph-paper distance, by Pythagoras.
What goes wrong when columns wear different units is also a later post's job.
For this post, watch the formula work on small numbers.
Here is a 2-column example, by pencil.
The new row has (rooms=4, income=3).
Compare it to one working row (rooms=5, income=3).
column a_j (new) b_j (work) gap_j = a_j - b_j gap_j^2
-----------------------------------------------------------------
rooms 4 5 -1 1
income 3 3 0 0
sum = 1
gap = sqrt(1) = 1
Now compare the same new row to another working row (rooms=8, income=9):
column a_j (new) b_j (work) gap_j = a_j - b_j gap_j^2
-----------------------------------------------------------------
rooms 4 8 -4 16
income 3 9 -6 36
sum = 52
gap = sqrt(52) ~ 7.21
The first working row has gap=1.
The second working row has gap=7.21.
So the first row is much closer than the second.
k is the count of neighbours we ask.
If k=1, the guess is the single closest row's price.
If k=3, we average the 3 smallest-gap rows' prices.
>> One clerk, one slate: Same new row (rooms=4, income=3). A third working row sits at (rooms=4,
income=7). Compute its gap before reading on.
check your slate: rooms 4-4 = 0, square 0; income 3-7 = -4, square 16;
sum 0 + 16 = 16; gap = sqrt(16) = 4. Of the three working rows the closest
is still the first (1 < 4 < 7.21) -- at k=1 the guess is that row's price.
This rule barely deserves the name "machine".
It builds nothing ahead of time.
It just keeps the whole working pile on a shelf.
When a new house walks in, it finds the rows most like it.
Then it averages what those rows sold for.
The one knob to set by hand is k, the count of neighbours to ask.
Small k makes the rule too jumpy.
Large k makes the rule too stiff.
The full derivation, and how to find the sweet spot, is in Part 2.
Count the clerk-steps for ONE guess, using all 8 columns this time.
Per working row: 8 subtractions, 8 squarings, 7 additions, 1 root.
That is 8 + 8 + 7 + 1 = 24 strokes per working row.
There are 16,512 working rows.
So one guess costs 16,512 x 24 = 396,288 strokes.
That is about four hundred thousand pencil strokes for a single new house.
A room of clerks does it by lunch.
But remember the bill: it comes due every time another house walks in.
MEASURING HOW WRONG: RMSE
A guesser is only as good as its misses are small.
So before trusting any rule we need an honest way to size up how wrong it is.
Define the leftover for one row.
Leftover r_i = y_i - yhat_i.
Here y_i is the row's true answer and yhat_i is the rule's guess for that row.
So the leftover is right answer minus guess.
RMSE = sqrt( (1/n) sum_i r_i^2 )
Read the symbols.
RMSE stands for root-mean-square-error.
n is the count of rows being scored.
sum_i means add over every row i.
So RMSE squares each leftover, averages the squares, then takes the square root.
Here is a 4-person worked example, by pencil.
person truth y guess yhat miss r r^2
--------------------------------------------------
A 1.4 1.2 +0.2 0.04
B 2.8 3.1 -0.3 0.09
C 0.9 1.0 -0.1 0.01
D 3.5 3.2 +0.3 0.09
sum r^2 = 0.23
mean r^2 = 0.23 / 4 = 0.0575
RMSE = sqrt(0.0575) ~ 0.24
RMSE = 0.24 is in units of $100,000s, so the typical miss is about $24,000.
The bar to beat is the FOOL who always shouts the average.
The fool ignores the columns and guesses the average price every time.
Work the fool out on the same 4 people.
His one guess is the average y = (1.4+2.8+0.9+3.5)/4 = 8.6/4 = 2.15.
He shouts that 2.15 four times:
person truth y fool's guess miss r r^2
----------------------------------------------------
A 1.4 2.15 -0.75 0.5625
B 2.8 2.15 +0.65 0.4225
C 0.9 2.15 -1.25 1.5625
D 3.5 2.15 +1.35 1.8225
sum r^2 = 4.3700
mean r^2 = 4.37 / 4 = 1.0925 fool's RMSE = sqrt(1.0925) ~ 1.05
Notice what we just computed for the fool.
We took squared gaps from the average, averaged them, then rooted.
That recipe is the SPREAD of the truth column itself.
Spread of a column is written std(y), the standard deviation of y.
So the fool's RMSE and std(y) are the same recipe.
Which means they cannot help but be the same number.
Set the clerks on the full 20,640-row column with these exact steps and the fool lands near 0.97.
Our rule scored RMSE = 0.24.
The fool scored RMSE = 1.05.
So 1.05 / 0.24 ~ 4, meaning our rule beats the fool about four times over.
Person E walks in: truth 2.0, our rule guesses 2.4. Add him to OUR table
(sum r^2 was 0.23 over 4 people) and recompute the rule's RMSE.
check your slate: miss r = 2.0 - 2.4 = -0.4; r^2 = 0.16; new sum = 0.23 +
0.16 = 0.39; mean = 0.39 / 5 = 0.078; RMSE = sqrt(0.078) ~ 0.28. One sloppy
guess dragged the typical miss from $24,000 up to $28,000 -- squaring makes big
misses shout.
RMSE punishes large single misses more heavily than a plain average of leftover sizes.
That plain average is MAE = mean|r_i|, the mean of the leftover sizes ignoring sign.
Check why RMSE punishes big misses harder.
Person B missed by -0.3, which squares to 0.3 x 0.3 = 0.09.
A double-sized miss of -0.6 squares to 0.6 x 0.6 = 0.36.
And 0.36 / 0.09 = 4, so doubling the miss gives it four times the weight.
RULE 2 -- THE STRAIGHT-STICK RULE
IN HAND: one guessing rule and one honest ruler.
The rule is ask-closest: measure gaps, then average the k nearest answers.
The ruler is RMSE: take each miss, square it, average, then root.
Now a second rule of the opposite temperament.
d+1 dials: [nudge b0, dial1 b1, dial2 b2, ...]
guess = b0 + b1*(rooms) + b2*(income) + ...
= X*beta (one dot-product per row)
Read the symbols.
d is the count of measured columns, so there is one dial per column plus one nudge, giving d+1 dials.
b0 is the fixed nudge added to every guess.
b1, b2, ... are the dials, one weight per column.
X*beta is a shorthand: multiply each column by its dial and add, for one row at a time.
Where the ask-closest rule hoards every row, this rule squeezes the whole working pile into just d+1 dials.
Then it throws the pile away.
From then on each guess is a single weighted sum.
The dials are not searched for.
They drop out of an exact formula.
Full derivation in Part 3.
PICKING SETTINGS WITHOUT PEEKING AT THE HIDDEN PILE
IN HAND: two rules, one ruler, and one drawer that must stay shut.
The first rule is ask-closest, with its knob k (the count of neighbours).
The second rule is straight-stick, with its d+1 dials (one weight per column plus a nudge).
The ruler is RMSE: take each miss, square it, average, then root.
The drawer is the hidden pile, the locked-away 20% we never peek at until the end.
Left to do: pick k without the drawer.
working pile split into 5 chunks: A B C D E
round 1 [#] . . . . -> mistake1
round 2 . [#] . . . -> mistake2
round 3 . . [#] . . -> mistake3
round 4 . . . [#] . -> mistake4
round 5 . . . . [#] -> mistake5
average -> honest estimate
Each round, one chunk grades the rule built on the other four.
Rotate until every chunk has graded exactly once.
That gives five mistake numbers, one per round.
Average those five mistakes.
This gives an honest estimate without ever touching the hidden pile.
PREPARE NUMBERS INSIDE EACH ROUND
Any prep that learns from the numbers (an average, a spread) must be done on each
round's building portion only. Doing it on the whole working pile before splitting
leaks grading-chunk numbers into the build. A bound pipeline (a chain that relearns
the prep each round) enforces this automatically; Part 2 shows the chain.
That single discipline is the spine the whole blog hangs on.
The discipline: never let the drawer (the hidden pile) influence a choice.
Here is the order of work.
First, pick settings using rotating folds on the working pile.
Then refit the winner on all of the working pile.
Then read the hidden pile ONCE.
Then report that number, and live with it.
>> NOTE: ISN'T CHECKING THE HIDDEN PILE PEEKING?
No. Using the hidden pile to pick among rules is peeking.
Using it ONCE to report the chosen rule's mistake is exactly what it is for.
A gap between the rotating-fold estimate and the hidden-pile mistake means the rule
memorised the working pile a little.
WHOLE BLOG IN SIX LINES
Here is everything above, boiled down to six lines.
Pin them somewhere.
Every later post is one of these lines, slowed down and worked out in full.
1. Find a rule that guesses right on rows never seen; measure on the hidden pile only.
2. Ask-closest rule -- store the pile, look it up; put columns on the same ruler;
pick k.
3. Straight-stick rule -- d+1 dials, exact formula, one dot-product per guess.
4. Measure mistake with RMSE (same units as price); the bar to beat is std(y).
5. Pick settings by rotating folds, preparing numbers inside each round.
6. Refit on all of working pile; read the hidden pile exactly once.
A handful of these bit me while writing the code. Each is a belief that feels
right and is not.
You reach for the size of the sheet and your hand wants one number:
20640 rows x 9 cols --> 185760 ?
But .shape never multiplies. It hands back the two counts side by side, rows
first, always:
df.shape == (20640, 9)
| |
rows cols rows = shape[0], cols = shape[1]
Read them apart; never fold them into one product.
Next: a shuffle cuts the pile differently every run, unless you pin its seed.
Pin a different seed in each place and your splits quietly drift apart:
random_state = 1 here --> split P
random_state = 42 there --> split Q P =/= Q, nothing lines up
So set one seed once at the top and reuse it -- same seed, same shuffle, your
splits match mine exactly: RANDOM_STATE = 42.
Two knobs are both small counts, so they get mistaken for one:
k = neighbours each guess consults (the ask-closest rule)
n_splits = chunks the working pile is cut into (the rotating-folds check)
k = 25 with 5 folds is no contradiction -- separate numbers for separate jobs,
and they never need to agree.
The scorer trips you the other way. Python's scoring is built to want BIGGER,
but a smaller RMSE is better -- the two pull opposite ways. So flip the sign
and let "bigger" mean "better":
RMSE: 0.66 0.74 (smaller is better)
score: -0.66 -0.74 (bigger is better -> -0.66 wins)
closer to 0 = smaller RMSE = best rule
Un-negate with -scores.mean() to read the real RMSE back.
Last one, and it is the dangerous one. Scale the whole working pile once and
THEN split for grading, and the grading rows have already leaked their centre
and spread into the build:
scale ALL --> split grading rows already seen (leak)
split --> scale each build grading rows stay unseen (clean)
A Pipeline chains scaler + rule so the scaler relearns fresh inside each fold's
build portion only.
Now that you have built the thing, here are the proper names.
Every post on this blog ends this way.
Plain words sit on the left; the textbook term sits on the right.
Practice the plain idea first.
The label is just a handle to carry it by.
Plain term used above Standard label
----------------------------------- ------------------------------------------
sheet of written-down numbers dataset / design matrix X
one measured column feature
right-answer column target / label y
working pile training set
hidden pile test set
store the pile, look it up non-parametric / instance-based
ask-closest rule k-nearest neighbours (KNN)
straight-stick rule linear regression (OLS)
dials weights / coefficients beta
fixed nudge intercept / bias beta0
put columns on the same ruler standardisation / normalisation
leftover / mistake (RMSE) residual / loss / error
rotating folds k-fold cross-validation
memorising the working pile overfitting
mistake on rows never seen generalisation error
Nothing above needed a computer -- only pencils, clerks, and patience. This last
section is for the day you meet one: the same six steps, spoken in Python.
>> NEW TO PYTHON? Each named once:
fetch_california_housing() -- loads the California sheet (no file needed)
train_test_split() -- carves off the hidden 20%
StandardScaler() -- puts every column on the same ruler
KNeighborsRegressor(k) -- the ask-closest machine; k sets how many neighbours
LinearRegression() -- the straight-stick machine
Pipeline([("name", step)]) -- chains scaler + rule so scaler relearns inside each fold
cross_val_score() -- rotating-folds scorer (returns negated RMSE; un-negate with -)
np.sqrt() -- square root
np.mean() -- plain average
np.std() -- spread of a column (same recipe as the fool's RMSE)
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split, cross_val_score, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
RANDOM_STATE = 42 # same seed everywhere so your splits match mine
The sheet X is 20640 rows by 8 columns; y is one price per row, lined up beside it:
X = row 0 [ 8.33 41 6.98 ... 37.88 ] -> y[0] = 4.526
row 1 [ 8.30 21 6.24 ... 37.86 ] -> y[1] = 3.585
... ...
row 20639 [ 2.39 16 5.25 ... 39.43 ] -> y[20639] = 0.894
\________ 8 columns ________/ one price per row
(20640 rows tall)
--- LOAD THE SHEET ---
data = fetch_california_housing()
X, y = data.data, data.target
print(X.shape) # (20640, 8) -- 20640 rows, 8 measured columns
print(y.shape) # (20640,) -- one price per row
--- CARVE OFF THE HIDDEN 20% ---
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=RANDOM_STATE)
print(X_train.shape[0]) # 16512 <- working pile
print(X_test.shape[0]) # 4128 <- hidden pile, locked away
the 20640 rows, cut 80/20:
[############### working pile: 16512 rows ###############|##### hidden: 4128 #####]
build the rule on these opened once, at the very end
--- THE ALWAYS-AVERAGE FOOL (BASELINE) ---
baseline_rmse = np.std(y_train)
print(baseline_rmse) # 1.1541 <- the bar to beat
--- ASK-CLOSEST RULE (K=25) ---
knn_pipe = Pipeline([
("scale", StandardScaler()), # put all 8 columns on the same ruler
("knn", KNeighborsRegressor(n_neighbors=25)) # ask the 25 closest rows
])
knn_pipe.fit(X_train, y_train)
knn_preds = knn_pipe.predict(X_test)
knn_rmse = np.sqrt(np.mean((y_test - knn_preds)**2))
print(knn_rmse) # 0.6308 <- typical miss ~$63,000; beats the fool by 1.83x
--- STRAIGHT-STICK RULE ---
lr_pipe = Pipeline([
("scale", StandardScaler()),
("lr", LinearRegression())
])
lr_pipe.fit(X_train, y_train)
lr_preds = lr_pipe.predict(X_test)
lr_rmse = np.sqrt(np.mean((y_test - lr_preds)**2))
print(lr_rmse) # 0.7270 <- typical miss ~$73,000; KNN wins here
--- PICK K BY ROTATING FOLDS (NEVER TOUCH THE HIDDEN PILE HERE) ---
COMPUTER RUNS THE SAME THREE LINES FOUR TIMES, ONCE PER K VALUE.
SHOWN FLAT SO YOU CAN SEE EVERY RUN -- THE LOOP IS JUST THE SAME WORK REPEATED:
cv = KFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)
K = 5
pipe = Pipeline([("scale", StandardScaler()), ("knn", KNeighborsRegressor(n_neighbors=5))])
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="neg_root_mean_squared_error")
print(-scores.mean()) # 0.7408 -- too jumpy: 5 neighbours copies noise
K = 10
pipe = Pipeline([("scale", StandardScaler()), ("knn", KNeighborsRegressor(n_neighbors=10))])
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="neg_root_mean_squared_error")
print(-scores.mean()) # 0.6945 -- better
K = 25 <- SMALLEST MISTAKE; PICK THIS ONE
pipe = Pipeline([("scale", StandardScaler()), ("knn", KNeighborsRegressor(n_neighbors=25))])
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="neg_root_mean_squared_error")
print(-scores.mean()) # 0.6612 -- best
K = 50
pipe = Pipeline([("scale", StandardScaler()), ("knn", KNeighborsRegressor(n_neighbors=50))])
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="neg_root_mean_squared_error")
print(-scores.mean()) # 0.6781 -- worse again: 50 neighbours blurs the detail
The four runs, as an elbow -- average fold-mistake down, k across:
0.74 | * k= 5 0.7408 too few: copies noise
0.70 | * k=10 0.6945
0.66 | * * k=25 0.6612 lowest -- pick this
| k=50 0.6781 too many: blurs detail
+----+-----+------------+-------- k
5 10 25 50
Down the left wall the mistake falls (more neighbours steadies the guess); past k=25
it climbs again (too many neighbours average in strangers). The elbow is k=25.
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 1 -- Predicting House Prices):
Part 1 (this post) .
Part 2 -- Ask-Closest Rule .
Part 3 -- Straight-Stick Rule
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================