==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 1 . PREDICTING HOUSE PRICES . PART 3 OF 3
The Straight-Stick Rule: Setting the Dials, Checking the Leftovers, Getting Numbers Right
============================================================================================
The ask-closest rule never really learned anything -- it just kept the answer key handy.
This rule is its opposite, and its better-behaved sibling. It actually studies the pile,
distils it down to a handful of dials, and then throws the pile away. From that point on
it carries no baggage: every guess is a quick weighted sum of the columns plus a fixed
nudge.
And here is the part that still feels like magic the first time you see it. The best
possible dials are not hunted for by trial and error -- they fall straight out of one
exact formula. No searching, no luck. This post derives that formula by hand on three
rows, then shows it is the very same arithmetic the toolbox runs on sixteen thousand.
To find the best dial you could try every value and keep the one with the
smallest total miss. But "every value" is endless -- 1.0, 1.001, 1.0001, on
forever. You would never finish.
Plot the total miss against the dial, though, and it is always the same shape: a
bowl with exactly one lowest point.
miss
\ /
\ /
\___ ___/
\___ ___/
\_______/
^
flat bottom, slope = 0 <- the best dial sits here, and only here
Down the left wall the curve slopes down; up the right wall it slopes up; at the
single lowest point the slope is exactly zero. So you do not hunt. You ask one
question -- at what dial is the slope zero? -- and algebra hands the answer back
in closed form. The formula you derive IS the bottom of the bowl.
THE RULE
one row of measured things: [rooms, income, age, ...]
x
one dial per column: [b1, b2, b3, ...]
+ fixed nudge b0
= guess yhat
all rows at once: yhat = X*beta
(X has a column of 1s prepended to carry b0)
yhat_i = b0 + sum_j beta_j * x_ij <=> yhat = X*beta
Each dial beta_j is the change in the guess per unit change in column j, all other
columns held fixed. The columns can be squares, logs, or products of raw columns -- the
rule stays linear in the dials.
SQUARED LEFTOVER TO SHRINK
\ /
\ / height = total squared leftover J(beta)
\ / bottom = the best dials beta
\ . /
Pick the dials that shrink the squared leftover sum:
J(beta) = ||X*beta - y||^2 = sum_i (yhat_i - y_i)^2
J is bowl-shaped -- its second-slope (Hessian) 2*X^T X is never-negative -- so any flat
point is the global bottom. No false bottoms, no searching.
FINDING THE DIALS: THE FLAT-POINT EQUATIONS
Set the slope of the bowl to zero:
grad J = 2*X^T (X*beta - y) = 0 => X^T X beta = X^T y => beta = (X^T X)^-1 X^T y
Worked by hand at one column (d = 1) on three rows. First, build the raw table:
x = 1, 2, 3 y = 2, 3, 5
row x y x^2 x*y
---------------------------
1 1 2 1 2
2 2 3 4 6
3 3 5 9 15
---------------------------
sum 6 10 14 23
X has a column of 1s for the nudge, plus the x column:
X = [ 1 1 X^T X = [ 3 6 X^T y = [ 10
1 2 6 14 ] 23 ]
1 3 ]
The system X^T X beta = X^T y:
3*b + 6*w = 10 (equation 1)
6*b + 14*w = 23 (equation 2)
Solve by elimination. Multiply equation 1 by 2:
6*b + 12*w = 20 (equation 1, doubled)
Subtract from equation 2:
(6*b + 14*w) - (6*b + 12*w) = 23 - 20
2*w = 3
w = 3/2 = 1.5 <- the dial on x
Put w back into equation 1:
3*b + 6*(3/2) = 10
3*b + 9 = 10
3*b = 1
b = 1/3 ~ 0.333 <- the fixed nudge
stick: yhat = 1/3 + (3/2)*x
yhat = 0.333 + 1.5*x
Check at each x:
x=1: yhat = 0.333 + 1.5*1 = 1.833
x=2: yhat = 0.333 + 1.5*2 = 3.333
x=3: yhat = 0.333 + 1.5*3 = 4.833
New pile (made-up), same recipe: x = 1, 2, 3 and y = 1, 3, 5. Build the four
sums, write the two flat-point equations, and solve for the dials.
check your slate: sum x = 1+2+3 = 6; sum y = 1+3+5 = 9; sum x^2 = 1+4+9 = 14;
sum x*y = 1 + 6 + 15 = 22. Equations: 3*b + 6*w = 9 and 6*b + 14*w = 22.
Double the first: 6*b + 12*w = 18; subtract: 2*w = 22 - 18 = 4, so w = 2; put
back: 3*b + 6*2 = 9, 3*b = 9 - 12 = -3, b = -1. Stick: yhat = -1 + 2*x, which
hits 1, 3, 5 exactly -- every leftover is zero; this stick runs straight through
all three points.
CHECKING THE LEFTOVERS
IN HAND: three rows (x = 1, 2, 3 against y = 2, 3, 5) and the dials the flat-point
solve dropped out: 2*w = 3 so w = 3/2 = 1.5, then 3*b = 10 - 6*1.5 = 10 - 9 = 1 so
b = 1/3 ~= 0.333. The stick reads yhat = 0.333 + 1.5*x. This section adds the test
every honest fit must pass: what its leftovers obey.
At the best dials, X^T (y - X*beta) = 0, written X^T e = 0: the leftover column sits at
right-angles to every measured column. Because one column is all ones, the leftovers
also sum to zero.
x=1: yhat=1.833, leftover = +0.167
x=2: yhat=3.333, leftover = -0.333
x=3: yhat=4.833, leftover = +0.167
sum leftover = 0 ok (right-angles to the ones column)
sum x * leftover = 0 ok (right-angles to x column)
RMSE, worked: the exact leftovers are 1/6, -1/3, 1/6; squares 1/36 + 4/36 + 1/36
= 6/36 = 1/6; mean = (1/6) / 3 = 1/18 ~= 0.0556; RMSE = sqrt(1/18) ~= 0.236
Verify the second right-angle check on your own slate: multiply each leftover by
its x and add. Use the exact fractions: leftovers 1/6, -1/3, 1/6.
check your slate: 1*(1/6) = 1/6; 2*(-1/3) = -2/3 = -4/6; 3*(1/6) = 3/6;
sum = (1 - 4 + 3)/6 = 0/6 = 0. The leftover column sits at right-angles to the
x column: the stick has already squeezed out everything x can explain.
Geometrically: yhat = H*y where the flat-shadow thrower H = X (X^T X)^-1 X^T projects y
onto the flat sheet spanned by the measured columns. H is symmetric and squaring it
gives itself back (H^2 = H). trace(H) equals the number of fitted dials -- here the
nudge b plus the dial w, 1 + 1 = 2. No point on the flat sheet sits closer to y.
SAME WORK, BIGGER SHEET
The three-row pencil solve is identical to what the toolbox runs on 16,000-odd rows
(the working pile: 20,640 x 8/10 = 16,512) and 8 columns -- only the size of X^T X
changes. Fitted dials land in m.coef_, the fixed
nudge in m.intercept_.
WHEN THE STRAIGHT-STICK RULE IS THE BEST POSSIBLE
The Gauss-Markov result: if the leftover has zero average given the rows (E[eps|X] = 0),
the same spread everywhere, leftovers do not echo each other, and X has full column rank
-- then the straight-stick rule is BLUE: the lowest-spread rule among all flat, honest
rules.
- Bell-curve shape of leftovers is NOT required for BLUE. You need it only for exact
confidence ranges and t-tests, not for the rule to be the best flat rule.
- Full column rank means no measured column is a recipe of the others. If one is,
X^T X is flat (no single solution) and (X^T X)^-1 does not exist.
GETTING THE DIALS RIGHT: DON'T FLIP THE SHEET
IN HAND: the exact formula beta = (X^T X)^-1 X^T y, proved bowl-bottomed and checked
by hand -- three rows gave w = 3/2 = 1.5 and b = 1/3 ~= 0.333, with leftovers
1/6 - 2/6 + 1/6 = 0 summing to zero. This section adds the bill for the solve, and
the safe way to run it.
build X^T X: O(n*d^2) <- n rows, each d^2 multiplications
solve for beta: O(d^3)
total: O(n*d^2 + d^3)
Pencil that bill in clerk-steps on our pile. With the ones column, d = 8 + 1 = 9, so
each row costs 9 x 9 = 81 multiplications; the working pile's 20,640 x 8/10 = 16,512
rows cost 16,512 x 81 = 1,337,472 strokes to build X^T X, and the solve adds about
9 x 9 x 9 = 729 more. A room of clerks clears it in a day or two -- ONCE. After that
every guess is 8 multiplications + 8 additions = 16 strokes: one clerk, one breath.
Do NOT compute (X^T X)^-1 by flipping the sheet directly. Forming X^T X squares the
shakiness number (kappa(X^T X) = kappa(X)^2) and loses precision. Instead:
- Solve the flat-point equations directly (Cholesky cut of X^T X), or better,
- Cut X itself with QR or SVD and solve without ever forming X^T X. The toolbox's
LinearRegression uses an SVD-based approach.
- For very many rows or a stream, roll downhill a little at a time: O(n*d) per step
instead of a full cut.
- If X^T X is flat or shaky (one column is a near-recipe of another), add a small bump
to the diagonal -- the ridge rule: beta = (X^T X + lambda*I)^-1 X^T y -- solvable
for any lambda > 0.
>> NOTE: SAME-RULER CONTRAST WITH ASK-CLOSEST RULE
The straight-stick rule does NOT care which ruler you used. Rescale a column and its
dial rescales inversely -- guesses and RMSE stay the same. (The ask-closest rule
breaks without same-ruler treatment.) You DO need same-ruler when rolling downhill or
when adding a bump, because those care about the size of the dials, which depends on
scale.
ASK-CLOSEST VS. STRAIGHT-STICK
IN HAND: both rules of this chapter, built and checked. Ask-closest keeps the whole
working pile (20,640 x 8/10 = 16,512 rows) and pays 16,512 x 24 = 396,288 strokes per
guess; straight-stick keeps 8 + 1 = 9 dials and pays 8 multiplications + 8 additions
= 16 strokes per guess. This section adds the two temperaments side by side.
Ask-Closest Rule (KNN) Straight-Stick Rule (OLS)
------------------ ------------------------ --------------------------
What it stores whole pile d+1 dials only
Build cost O(1) (store only) O(n*d^2 + d^3) (solve)
Guess cost O(n*d) per row O(d) per row
Setting by hand k (by rotating folds) none (lambda if adding bump)
Same-ruler needed? yes -- required no -- not for plain rule
Guess surface piecewise-flat, local one flat sheet, global
Two rules, opposite temperaments, one question: given the working rows, guess the right
answer for a new one. The ask-closest rule leans on stored neighbours and does its
thinking at the last second; the straight-stick rule does its thinking up front and
compresses everything into one flat sheet. Neither is "the" answer -- which to reach for
depends on the pile in front of you, and that judgement is the real craft.
GOING DEEPER
- flat-point equations, ordinary least squares -- the solve
- QR / SVD least squares, shakiness number (condition number) -- the numerics
- flat-shadow thrower (hat matrix), right-angle leftovers -- the geometry
- Gauss-Markov, uneven miss spread -- the assumptions
- ridge, lasso -- when columns are near-recipes of each other
- The rule is yhat = X*beta (a guess is a weighted sum of the measured columns plus a
fixed nudge); set the dials by shrinking the squared leftover ||X*beta - y||^2.
- That squared leftover is bowl-shaped, which means its one flat point is the global
bottom -- therefore the best dials drop out of the exact formula beta = (X^T X)^-1 X^T y,
no searching.
- At those dials the leftover column sits at right-angles to every measured column, so
the guess yhat = H*y is just the shadow of y thrown flat onto the sheet the columns span.
- That shadow is the lowest-spread honest flat guess (BLUE) when the Gauss-Markov terms
hold; the bell-curve shape of leftovers is needed only for confidence ranges and t-tests.
- Because forming X^T X squares the shakiness, cut the sheet directly (QR or SVD) instead
of flipping it; and when X^T X is flat (one column is a near-recipe of another) add a
small bump to the diagonal.
A few places that bit me, each one a thing that looks right.
MSE looks like it ought to end in a square root. It does not:
miss -> square -> mean -> STOP MSE
miss -> square -> mean -> sqrt RMSE (the root is the R)
Tack ** 0.5 onto mean_squared_error and you have quietly built RMSE instead. Read
the name -- the R is the root.
"Leftover = 2, so 2 is the stick's score" runs backwards. The leftover is the
misery STILL there after the stick tried:
total wobble = good part + leftover
(explained) (still missed)
R^2 = (total wobble - leftover) / total wobble <- the GOOD part's share
And R^2's bottom is the whole wobble, never the row count:
WRONG: (total wobble - leftover) / 5 (the 5 rows)
RIGHT: (total wobble - leftover) / total wobble
Dividing by wobble is what makes the units cancel -- which is exactly why R^2 has
none.
Add a column of pure noise, watch R^2 tick UP, and call the machine better? Only on
the TRAINING pile:
training R^2: add any column -> can only stay flat or rise (old fit still there)
test R^2: add noise -> usually FALLS
Judge on the test R^2 (or adjusted R^2); training R^2 always rewards more columns.
R^2 below 0 looks like a bug, but on unseen rows it is real:
train: the right-angle fit forces leftover <= wobble -> R^2 in [0, 1]
test: no such guarantee -> leftover CAN exceed wobble -> R^2 = 1 - RSS/TSS < 0
A negative R^2 just means the stick did worse than guessing the flat average.
Hunting the hardest-dragging column, -0.04 against -0.007:
-0.007 ----|--- 0
-0.04 -----| further below zero = SMALLER = drags hardest
-0.04 is the more-negative one, even though "4" feels smaller than "7". Use
min(dials, key=dials.get) to fetch that column's NAME.
Last, two raw dials cannot be ranked head to head when their columns wear
different units:
-0.007 per POUND (a pound is a tiny step)
-0.04 per HORSEPOWER (a horsepower is a big step)
Stand every column on one shared ruler (StandardScaler) first; then each dial reads
"answer-units per one-spread step" and the comparison is finally fair.
Nothing above needed a computer -- only pencils, clerks, and patience. This last
section is for the day you meet one: the same three rows, spoken in Python.
First the flat-point solve by hand -- three rows, four sums, the 2x2 system, all written
out, no loop, so you watch w=1.5, b=0.333, and the leftovers print back:
THREE ROWS, ONE COLUMN: (X, Y) ONE PER LINE
x1, y1 = 1, 2
x2, y2 = 2, 3
x3, y3 = 3, 5
FOUR SUMS THE FLAT-POINT EQUATIONS NEED
sum_x = 1 + 2 + 3 # 6
sum_y = 2 + 3 + 5 # 10
sum_xx = 1*1 + 2*2 + 3*3 # 14
sum_xy = 1*2 + 2*3 + 3*5 # 23
FLAT-POINT EQUATIONS (N=3): 3*B + 6*W = 10 AND 6*B + 14*W = 23
ELIMINATE B AND SOLVE THE 2X2 SYSTEM:
w = (3*sum_xy - sum_x*sum_y) / (3*sum_xx - sum_x*sum_x) # 9/6 = 1.5 (the dial on x)
b = (sum_y - w*sum_x) / 3 # (10-9)/3 = 0.333 (the nudge)
print(round(w,3), round(b,3)) # 1.5 0.333
STICK'S GUESS AT EACH X, AND THE LEFTOVER (Y - GUESS), ONE LINE EACH
yhat1 = b + w*1; yhat2 = b + w*2; yhat3 = b + w*3
print(round(yhat1,3), round(yhat2,3), round(yhat3,3)) # 1.833 3.333 4.833
print(round(y1-yhat1,3), round(y2-yhat2,3), round(y3-yhat3,3)) # 0.167 -0.333 0.167
Three rows, every sum and leftover visible. The real pile is 16,512 rows of 8 columns;
there X^T X is a 9x9 grid you cannot invert by pencil, so the toolbox runs the IDENTICAL
flat-point solve (via SVD) for you. Same arithmetic, bigger grid -- no setting to tune,
no rotating-fold hunt, the dials drop straight out:
>> NEW TO PYTHON? One new piece here:
thing.coef_ -- a value the object stores after being fitted; sklearn marks
every learned-from-data value with a trailing underscore
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
m = LinearRegression() # SVD-based flat-point solve under the hood
m.fit(X_train, y_train) # sets dials: X^T X beta = X^T y
pred = m.predict(X_test) # one dot-product per row
rmse = mean_squared_error(y_test, pred) ** 0.5
print(round(rmse, 3), m.coef_, m.intercept_)
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 1 -- Predicting House Prices):
Part 1 -- The Full Picture .
Part 2 -- Ask-Closest Rule .
Part 3 (this post)
Next chapter: Chapter 2 -- Grading a Guesser (MSE, R^2, reading the dials)
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================