==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 4 . HUMBLE DIALS AND WOBBLE BANDS . PART 1 OF 3
The Leash: Ridge, Lasso, and Humbling the Dials
============================================================================================
Back to guessing a number -- but this time the machine has a character flaw.
We are going to fix it.
First, the picture you need, built from scratch.
A "straight-stick rule" is a guess made by multiplying each measurement by its own number
and adding the results.
Each of those numbers is a "dial" -- a multiplier the machine turns up or down.
The machine sets each dial to make its misses as small as possible.
Until now we let those dials spin as far as they liked.
That freedom is the flaw.
Left off the leash, a dial will grow to a monster just to twitch through one more point in
the study pile -- the patients the machine learns from.
Which means a machine that bends itself around every point it has seen falls flat on its
face the moment it meets someone new.
This chapter is about humility, taught two ways.
Part 1 (this post) puts the dials on a LEASH so none of them can grow into a tyrant.
Part 2 asks a harder question -- how much should you even TRUST a dial? -- and answers it
by faking a hundred trips to the hospital.
Part 3 lifts the hood and shows where the dials come from in the first place.
Suppose two columns secretly say the very same thing -- weight in pounds, and
weight in ounces. One is just sixteen times the other, so they move in lockstep:
pounds : 10 20 30
ounces : 160 320 480 always 16x -- not one new fact between them
Now the machine is sitting on a loose seesaw. Whatever it leans onto pounds it
can take back off ounces, and the guess never moves:
dial_pounds = +1,000,000 term: +10,000,000
dial_ounces = -62,500 term: -10,000,000
sum: 0 (and 0 either way)
Both dials can run off to a billion and back while the prediction sits perfectly
still. The answer is stable; the dials are wildly unstable -- and unstable dials
are fragile dials, ready to swing on the next scrap of noise.
Ridge bolts the seesaw down by taxing big dials: every dial is pulled toward
zero, so neither twin can bloat. Lasso goes harder and drives one twin's dial to
exactly zero -- keeping one column and deleting the other outright.
A NEW SHEET: THE DIABETES PILE
diabetes -- 442 people, 10 body measurements, one answer
patient age bmi bp s1 ... s6 | answer
--------------------------------------------------
#1 0.04 0.06 0.02 ... | 151 <- disease score one YEAR later
#2 -0.01 -0.05 -0.03 ... | 75
... | ...
+--------------- 10 columns -----------+| the answer
The job: read a person's ten measurements, guess their number.
Here that number is a disease progression score taken a year later.
The straight-stick rule -- multiply each measurement by its own dial, add the results,
add one fixed lift -- gives the guess:
guess = (dial1 * age) + (dial2 * bmi) + ... + (dial10 * s6) + nudge
The "nudge" is that one fixed lift added at the end, the same for every patient.
The dials are still whatever makes the total squared miss smallest.
("Squared miss" = take each miss, the answer minus the guess, and multiply it by itself,
so a plus miss and a minus miss cannot cancel.)
Nothing new yet.
What is new is that we no longer trust the machine to set those dials sensibly on its own.
DISEASE: HUGE DIALS THAT MEMORISE
Here is the failure, drawn. Give the dials free rein and they grow huge to chase every
little wiggle in the study pile:
small honest dial: huge memorising dial:
line follows the trend line whips up and down to touch every dot
. . . .
. . . \ . /\ . /
. . . . .\/ \/ .
smooth, travels well perfect on study, WILD on anyone new
A huge dial means the machine has fallen in love with one column and is over-trusting it.
Which means it nails the people it has already seen.
So it scores beautifully on the study pile.
But then it shatters on a new patient.
That is memorising, not learning.
There is a name for this failure: the machine is "too jumpy."
A too-jumpy machine swings wildly when the data shifts even a little.
Its opposite is "too stiff" -- a machine so rigid it ignores real patterns.
Good machines sit between the two, and huge dials are the jumpy extreme.
So we want to FORBID huge dials.
LEASH: ADD A FINE
Until now the machine paid only for being wrong.
Now we make it pay a second fine -- for being immodest.
So it must drive down two things at once:
total wrongness + a * (a fine for big dials)
+- fit the data -+ +------- stay humble -------+
The letter a (the textbooks write the Greek "alpha") is a knob WE set by hand.
It says how hard to press for humility.
a = 0 no fine -- the old free-spinning machine, back to memorising
a small gentle squeeze
a huge dials crushed toward nothing -- maybe too stiff now
There are two famous fines. On paper they look almost like twins. At the bottom, where it
counts, they behave completely differently -- and that one difference is the whole point
of this post.
RIDGE: THE SMOOTH SQUEEZE
IN HAND: the diabetes sheet -- 442 people, 10 columns, one answer.
The guess is the straight-stick rule: 10 dials plus 1 nudge = 11 numbers.
The bill has two charges: total wrongness, plus a times a fine for big dials.
("a" is the knob WE set for how hard to press for humility.)
This section gives the fine its first shape: the square.
Ridge's fine is the SQUARE of each dial, all of them added up.
("Square" = the dial multiplied by itself.)
make smallest: (add up every miss^2) + a * (add up every dial^2)
To feel what a square does, picture a tug-of-war over a single dial.
Its usefulness pulls it away from zero.
The fine pulls it back toward zero.
How hard does the fine pull?
For a square, the pull is 2*a*dial.
Which means as the dial gets SMALLER, the fine's grip gets weaker too.
So the grip fades to nothing at zero:
Ridge -- a smooth bowl
fine | . .
| . .
| . .
| .___. <- almost flat at the bottom: no force left near zero
+---------------------- dial size
0
So the tug-of-war settles at a dial that is SMALL but not zero.
The fine drags every dial inward.
But near zero it runs out of strength.
So no dial is ever shoved all the way to exactly zero.
Ridge shrinks the whole orchestra; it silences nobody.
And because that bowl is smooth, there is a clean one-shot answer -- no searching.
Write the sheet as a grid X (people down, columns across) and the answers as a column y.
Then the ridge rule for all the dials at once is:
dials = (X^T X + a*I)^-1 X^T y
Here X^T means X flipped on its side (rows become columns).
X^T X is the grid that records how every column overlaps every other column.
(...)^-1 means the inverse of that grid -- the "untangler" that shares credit between
columns that move together.
I is the identity grid (1's down the diagonal, 0's elsewhere).
So a*I just adds a BUMP of size a down the diagonal before the solve.
For one lonely dial that rule reads as the tidy picture:
Ridge dial = top / (bottom + a) bigger a -> bigger bottom -> smaller dial
What are "top" and "bottom"? Real numbers from the sheet:
top = how the column and the answer move together
= sum of (column value - column average) x (answer - answer average)
bottom = how much the column wiggles on its own
= sum of (column value - column average)^2
Here "column" means one measurement across all the people (say bmi).
"Column average" means add that column's values and divide by how many people.
"Answer average" means add the answers and divide by how many people.
For ONE column (say bmi), ONE dial, on 4 people:
person bmi answer bmi-avg ans-avg (bmi-avg)*(ans-avg) (bmi-avg)^2
--------------------------------------------------------------------------------
A 0.04 97 -0.06 -29 1.74 0.0036
B 0.06 121 -0.04 -5 0.20 0.0016
C 0.12 135 0.02 9 0.18 0.0004
D 0.18 151 0.08 25 2.00 0.0064
avg: 0.10 126 ------ -------
top=4.12 bottom=0.0120
without ridge (a=0): dial = 4.12 / 0.0120 = 343
with ridge a=10: dial = 4.12 / (0.0120 + 10) = 4.12 / 10.0120 = 0.411
with ridge a=100: dial = 4.12 / (0.0120 + 100) = 4.12 / 100.0120 = 0.0412
Read those three lines one at a time.
With a=0 the bottom is just 0.0120, so the dial is 4.12/0.0120 = 343 -- huge.
With a=10 the bottom becomes 0.0120+10 = 10.0120, so the dial is 4.12/10.0120 = 0.411.
That is 343/0.411 = 834 times smaller.
With a=100 the bottom becomes 0.0120+100 = 100.0120, so the dial is 4.12/100.0120 = 0.0412.
That is 343/0.0412 = 8325 times smaller.
So the bigger the a, the smaller the dial.
Now the table's own arithmetic, spelled out.
The table uses a column average of 0.10 and an answer average of 126.
"bmi-avg" means bmi minus the column's own average (0.10).
So for person A: 0.04 - 0.10 = -0.06, and for B: 0.06 - 0.10 = -0.04, and so on.
"ans-avg" means answer minus the answer's average (126).
So for person A: 97 - 126 = -29, and for B: 121 - 126 = -5, and so on.
Multiplying those two gives the "top" piece for each person.
Adding up all four "top" pieces gives 1.74 + 0.20 + 0.18 + 2.00 = 4.12.
Squaring each "bmi-avg" gives the "bottom" piece for each person.
Adding up all four "bottom" pieces gives 0.0036 + 0.0016 + 0.0004 + 0.0064 = 0.0120.
Same column (top = 4.12, bottom = 0.0120). Set the knob to a = 1 and work the
ridge dial on the slate.
check your slate: dial = top / (bottom + a) = 4.12 / (0.0120 + 1) = 4.12 /
1.0120 ~= 4.07. Even a gentle a = 1 drags the plain dial of 343 down to about
4.1 -- the bump on the bottom does nearly all the work.
RIDGE SHRINKS ALL, ZEROES NONE, AND SOLVES IN ONE SHOT
The fine's pull dies at zero, so every dial lands small-but-alive.
The bump-on-the-diagonal makes the solve exact.
As a bonus it also cures a crash that plain least-squares can hit: when two columns
carry almost the same information, the untangler grid cannot be inverted and the solve
blows up. Adding a*I down the diagonal always makes the grid invertible again.
LASSO: THE SHARP SNAP
Lasso's fine is the ABSOLUTE size of each dial -- its size with the sign thrown away.
("Absolute size" of -0.7 is 0.7; the absolute size of 0.7 is also 0.7.)
make smallest: (add up every miss^2) + a * (add up every |dial|)
Same tug-of-war, one key change: the fine's pull is now CONSTANT.
For an absolute size the pull is just a, no matter how tiny the dial gets.
So it does not fade near zero.
It keeps shoving at full strength right up to the edge:
Lasso -- a sharp corner
fine | \ /
| \ /
| \ /
| \ /
| \ /
| \ /
| V <- a sharp point: full force right up to zero
+---------------------- dial size
0
Now the ending changes.
Even when a dial is already tiny, the fine still pushes with full strength a.
If that column's usefulness cannot beat a, the dial gets driven all the way to exactly zero.
And it is PINNED there.
strong column (usefulness > a): survives, just shrunk
weak column (usefulness < a): SNAPPED to exactly 0 -> dropped from the sheet
So Lasso does more than shrink -- it DELETES useless columns by zeroing their dials.
Which means it hands you back a shorter sheet.
And with it, an answer to a question Ridge never touches: which measurements actually
mattered?
The price of that sharp corner: there is no clean formula.
So you cannot solve it in one shot.
The machine has to loop:
1. start every dial at 0
2. nudge the bmi dial -- does its usefulness beat the fine? keep it : snap to 0
3. nudge the age dial -- same test
4. ... keep looping over all the dials until they stop moving
At the end you COUNT THE SURVIVORS -- the dials that did not snap to zero.
LASSO SHRINKS, SNAPS WEAK DIALS TO ZERO, AND HAS NO FORMULA
Because the fine never lets up, Lasso zeroes out weak columns entirely.
So it tells you which columns earned their keep.
The corner that buys that gift also costs the clean solve.
Which means Lasso must loop until the dials stop moving.
RIDGE VS LASSO, SIDE BY SIDE
Ridge Lasso
fine a * sum(dial^2) a * sum(|dial|)
shape smooth bowl sharp V-corner
pull near 0 fades to nothing stays at full strength
weak dials small but never 0 SNAP to exactly 0
columns keeps all 10 drops the useless ones
by hand yes: top/(bottom+a) no: must loop
gives you a humbler machine a humbler machine PLUS a shortlist
The ONLY real difference is square versus absolute.
That single choice decides whether the fine's pull dies at zero or stays alive.
Ridge squares the dial, so the pull dies at zero, so dials settle near zero but never reach it.
Lasso takes the absolute size, so the pull stays alive, so weak dials snap to exactly zero.
Everything else follows from that one choice.
PICKING THE KNOB A -- NEVER GUESS IT
Both machines have the same knob a.
You do NOT eyeball it.
Instead you let a "5-slice check" do the choosing.
Here is that check from scratch.
Cut the study pile into 5 equal slices.
Cover one slice, build the dials on the other four, guess the covered slice, write down the
error.
Rotate until every slice has had its turn as the covered one.
Average the five errors.
That average is one a-value's score.
Do this for a whole ladder of a-values -- about fifty of them -- and a valley appears:
error | \ _/ too-small a -> still memorising
| \_ ___/ too-big a -> dials crushed, too stiff
| \__ __/
| \__.__/ <- the valley = the best a
+--------------------------- a (from tiny to huge)
Pick the a at the bottom of the valley.
Then REBUILD the dials on the full study pile with that winning a.
Then grade ONCE on the sealed exam pile -- a held-back set of patients the machine never
touched while choosing a.
The iron rule: never pick a by peeking at that exam pile.
>> NOTE: THE CURVES ARE THE ANSWER, NOT A PLOT
The lab asks you to report the error-versus-a curve for both machines.
The plot is optional; the numbers behind it are the point.
The winning a is wherever the curve bottoms out -- the valley above, turned into a row
of numbers.
One candidate a was scored by the 5-slice check; its five covered-slice errors
came out 52, 48, 55, 50, 45 (made-up). Turn them into that a's one score.
check your slate: sum = 52 + 48 + 55 + 50 + 45 = 250; score = 250 / 5 = 50.
That single 50 is one dot on the valley curve; the winning a is wherever the
row of such dots bottoms out.
A few wrong pictures I had to untangle, each one a thing that feels right.
A 95% band sounds like it chops 25% off each end -- borrowing from how quartiles
cut the middle 50%. It does not. 95% means KEEP the middle 95%, so chop 2.5% off
each end:
|xxxxx|============== keep the middle 95% ==============|xxxxx|
2.5% 2.5%
q0.025 q0.975
For 200 values, 2.5% of 200 = 5: drop the 5 smallest and 5 largest, keep index
5..194. Reach for 0.25 instead of 0.025 and the band comes out 10x too narrow.
Hand RidgeCV a custom splitter, then read its stored curve, and it crashes:
cv=kf + later ridge_cv.cv_values_ -> AttributeError (nothing was stored)
RidgeCV keeps cv_values_ only when it runs its own built-in leave-one-out. Give it
cv=kf and it quietly stops storing them. Pass store_cv_values=True and let it use
its default instead. (LassoCV has no such quirk.)
A 95% band always keeps the same COUNT of notes, so the count says nothing -- the
WIDTH is the whole story:
bmi-dial: 0.42 .. 0.58 -> width 0.16 tight, trustworthy
s5-dial: -0.80 .. +0.90 -> width 1.70 wide, crosses 0, useless
\___ both keep exactly 190 of 200 notes ___/
Same 190 notes, opposite trust. The width is the gap between the 5th and 195th
note, never the count.
np.isin marks the people you PICKED. Grade on those and you test the machine on the
very patients it studied:
np.isin(...) -> True for the picked grading here = open-book = a lie
~np.isin(...) -> True for the NOT-picked grading here = honest
The ~ flips picked to not-picked. That one squiggle is the whole difference between
an honest score and an over-confident one.
Last, the per-column slope, sum((x-xbar)(y-ybar)) / sum((x-xbar)^2), looks like the
true dial. It is the true dial ONLY when no column overlaps another:
columns independent: ladder slope == real dial
columns overlap (bmi, bp, serums drift together): ladder slope =/= real dial
Real columns always overlap, so the true dial needs the untangler (X^T X)^-1 X^T y
to share credit between columns that move together. Call the lone slope a LADDER --
a first picture, a crutch -- never "the dial".
Plain term used above Standard label
----------------------------------- ------------------------------------------
the leash / a fine for big dials regularisation
Ridge -- the square fine L2 regularisation / ridge regression
Lasso -- the absolute fine L1 regularisation / lasso regression
the knob a (alpha) regularisation strength (alpha / lambda)
bump on the diagonal (X^T X + alpha*I)^-1 X^T y
snap a weak dial to zero feature selection / sparsity
count the survivors number of non-zero coefficients
the 5-slice check k-fold cross-validation
the valley the CV error curve / bias-variance sweet spot
too stiff vs too jumpy the bias-variance trade-off
Nothing above needed a computer -- only pencils, clerks, and patience. This last
section is for the day you meet one: the same steps, spoken in Python.
First the Ridge dial formula from the worked example, hard-coded -- one person per line:
RIDGE DIAL: 4-PERSON BMI EXAMPLE
A_bmi, A_ans = 0.04, 97
B_bmi, B_ans = 0.06, 121
C_bmi, C_ans = 0.12, 135
D_bmi, D_ans = 0.18, 151
bmi_avg = (A_bmi + B_bmi + C_bmi + D_bmi) / 4 # 0.10
ans_avg = (A_ans + B_ans + C_ans + D_ans) / 4 # 126
print(bmi_avg, ans_avg) # 0.1 126
TOP = SUM((BMI - BMI_AVG) * (ANS - ANS_AVG))
top_A = (A_bmi - bmi_avg) * (A_ans - ans_avg) # (-0.06)*(-29) = 1.74
top_B = (B_bmi - bmi_avg) * (B_ans - ans_avg) # (-0.04)*(-5) = 0.20
top_C = (C_bmi - bmi_avg) * (C_ans - ans_avg) # (0.02)*(9) = 0.18
top_D = (D_bmi - bmi_avg) * (D_ans - ans_avg) # (0.08)*(25) = 2.00
top = top_A + top_B + top_C + top_D # 4.12
print(round(top,2)) # 4.12
BOTTOM = SUM((BMI - BMI_AVG)^2)
bot_A = (A_bmi - bmi_avg)**2 # (-0.06)^2 = 0.0036
bot_B = (B_bmi - bmi_avg)**2 # (-0.04)^2 = 0.0016
bot_C = (C_bmi - bmi_avg)**2 # (0.02)^2 = 0.0004
bot_D = (D_bmi - bmi_avg)**2 # (0.08)^2 = 0.0064
bottom = bot_A + bot_B + bot_C + bot_D # 0.0120
print(round(bottom,4)) # 0.012
RIDGE DIAL = TOP / (BOTTOM + A) -- WATCH HOW A SHRINKS THE DIAL
dial_a0 = top / (bottom + 0) # 4.12/0.012 = 343 (plain, no leash)
dial_a10 = top / (bottom + 10) # 4.12/10.012 = 0.411
dial_a100 = top / (bottom + 100) # 4.12/100.012 = 0.0412
print(round(dial_a0), round(dial_a10,3), round(dial_a100,4)) # 343 0.411 0.0412
The bigger the leash a, the smaller the dial -- but never exactly zero. The toolbox
block below sweeps 50 values of a and keeps the valley:
Two ready-made machines do all of this: RidgeCV and LassoCV both sweep a ladder of a's
with the 5-slice check and keep the winner. Scale the columns first, the same as every
chapter before.
>> NEW TO PYTHON? Each named once:
np.logspace(-3, 3, 50) -- 50 values from 10^-3 to 10^3, spaced evenly in powers
Model(...).fit(X, y) -- build the machine and set its dials on the study pile
model.coef_ != 0 -- a yes/no for each dial: did it survive? (True = non-zero)
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
RIDGE: SWEEP 50 A'S, KEEP THE VALLEY. STORE_CV_VALUES LETS US READ THE CURVE LATER.
ridge_alphas = np.logspace(-3, 3, 50)
ridge_cv = RidgeCV(alphas=ridge_alphas, store_cv_values=True)
ridge_cv.fit(X_train_scaled, y_train)
ridge_best_alpha = float(ridge_cv.alpha_)
LASSO: ITS OWN A-LADDER, 5 FOLDS, MORE LOOP-ROOM (MAX_ITER) FOR THE SNAP TO SETTLE.
lasso_alphas = np.logspace(-3, 1, 50)
lasso_cv = LassoCV(alphas=lasso_alphas, cv=5, max_iter=10000)
lasso_cv.fit(X_train_scaled, y_train)
lasso_best_alpha = float(lasso_cv.alpha_)
n_nonzero_lasso = int(np.sum(lasso_cv.coef_ != 0)) # the survivors
ERROR-VS-A CURVES THE LAB ASKS FOR
ridge_cv_mse_mean = ridge_cv.cv_values_.mean(axis=0)
lasso_cv_mse_mean = lasso_cv.mse_path_.mean(axis=1)
RIDGECV QUIETLY DROPS THE CURVE IF YOU HAND IT A CUSTOM CV
RidgeCV will only save its error curve (cv_values_) when it uses its own built-in
leave-one-out check. The moment you pass it a custom splitter -- cv=kf -- it silently
stops storing those values, and the later line ridge_cv.cv_values_ crashes with an
AttributeError. The fix is to NOT pass cv=kf here; pass store_cv_values=True and let
RidgeCV use its default. (LassoCV has no such quirk -- cv=5 is fine there.)
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 4 -- Humble Dials and Wobble Bands):
Part 1 (this post) .
Part 2 -- One Dial Is a Lie .
Part 3 -- The Dial by Hand
Previous chapter: Chapter 3 -- Sorting Into Bins
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================