==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 3 . SORTING INTO BINS . PART 3 OF 4
Leash and Cloud: L2 Punishment and the Two-Cloud Wall
============================================================================================
An over-confident machine is a dangerous thing.
The machine we use here is the S-curve machine.
It keeps one DIAL per input column (a dial is just a number it can turn up or down).
For one row it multiplies each column by its dial and adds the products into one number.
Call that one summed number the dial sum z.
Then it SQUASHES z through the sigmoid: chance = 1 / (1 + e^-z).
Squash here means sigmoid -- the S-shaped curve that bends any z into a 0-to-1 chance.
So the machine turns a row into a single chance between 0 and 1.
This machine has a streak of over-confidence.
It sets its dials by rolling downhill on a leftover score.
That score is CROSS-ENTROPY: for each lump, read off the chance the machine gave the
TRUE bin, then fine it by -log of that chance.
Which means a confident wrong chance costs a lot and a hedged chance costs little.
Rolling downhill means nudging each dial in the direction that lowers that score.
But left to itself nothing stops one dial from ballooning to 50 to make the study pile
fit.
A dial that large means the machine has fallen for one column and stopped listening to
the other twenty-nine.
So it aces the practice exam and then freezes on a lump it has never seen.
Confident, and wrong, which is the worst way to be.
This post is about humbling it, two different ways.
The first keeps the same S-curve machine but puts it on a TIGHTER leash.
The second throws that machine out and tries a completely different temperament.
This second one never rolls downhill at all.
Instead it steps back, looks at the SHAPE of the two groups, and draws the wall in a
single stroke.
>> NOTE: THE PLAIN S-CURVE MACHINE ALREADY HAS A LEASH
Worth saying up front, because it surprises people: LogisticRegression() in sklearn is
NOT a free machine.
A "leash" here means a price the machine pays for big dials -- this is L2, where the
fine is the sum of every dial squared, which pushes dials toward zero.
The strength of that leash is set by C, where C = 1/lambda (small C = strong leash).
sklearn's defaults are penalty='l2', C=1.0 -- so the plain machine already carries an
L2 leash of medium strength.
This post is therefore not "no leash vs leash".
It is MEDIUM LEASH (C=1.0) vs TIGHTER LEASH (C=0.1).
To see a genuinely unleashed machine you would have to pass penalty=None explicitly.
One column, radius, predicts well on its own. So the machine falls for it and
cranks that one dial sky-high:
dial_radius = 40 every other dial ~ 0
verdict = 40 * radius + (almost nothing else)
Now the whole machine is a single column wearing a crowd. A new patient shows up
with a slightly odd radius and the prediction lurches the full width of the scale
on that one reading:
radius 1.0 --> chance 1%
radius 1.2 --> chance 99% a hair of input, the verdict flips
A machine balanced on one column is a fragile machine. So put it on a leash: tax
every large dial (the L2 penalty, the SVM's C). With big dials made expensive, it
can no longer afford a single giant -- it has to spread its bet across many
columns, and no lone outlier can flip the verdict by itself.
WHAT A LOOSE LEASH LETS HAPPEN
default machine (C=1.0): medium leash -> dials kept modest, but a dominant
column can still pull its dial fairly large
truly free (penalty=None): no leash -> one dial might reach 40 or -60
patient row -> *dials -> add -> squash -> chance
if column 7's dial = 40:
a shift of 0.1 in column 7 swings the sum by 40 x 0.1 = 4.0
-> chance jumps from 0.1 to 0.98
the machine bets almost everything on one column
-> cocky on study pile, shaky on new lumps
So the question this post really asks: the default C=1.0 leash is fine, but what happens
if we pull it TIGHTER (down to C=0.1, a stronger leash)?
Do the dials shrink further?
And does the machine get more humble?
A LEASH ON THE DIALS (L2 PENALTY)
Add a price tag to large dials.
Every extra unit of dial size costs something.
Which means the machine now minimises two things at once.
It must fit the study pile well AND keep the dials small.
L2 here is the name of that price: the fine is the sum of every dial squared.
The penalised objective is (beta_j is the dial on column j):
L_L2(beta) = L(beta) + lambda * sum_j beta_j^2
The first term L(beta) is the plain-fit score, the CROSS-ENTROPY.
Cross-entropy means: for each lump, read the chance the machine gave the TRUE bin, then
fine it by -log of that chance.
So a confident wrong chance costs a lot and a hedged one costs little.
The second term is the sum of squared dials times a strength lambda.
lambda is just a knob: bigger lambda means a harder squeeze on the dials.
When lambda is large, even a small sum-of-squared-dials gets expensive.
Therefore the machine is forced to shrink everything toward zero.
The gradient (the slope that says which way to nudge each dial) picks up a pull-toward-
zero term:
dL_L2/dbeta_j = (1/n) sum_i ( sig(z_i) - y_i ) x_ij + 2*lambda*beta_j
Here sig(z_i) is the squashed chance for row i and z_i is its dial sum.
The term 2*lambda*beta_j pulls every dial toward zero at every step.
So the dials spread across all 30 columns instead of concentrating on one.
>> NOTE: BAYESIAN READING
Adding a squared-dial penalty is the same as placing a zero-mean Gaussian prior on
each dial and maximising the posterior instead of the likelihood.
A prior here is a belief held before seeing data; zero-mean means we expect each dial
to sit near zero.
The prior says "a dial as large as 40 is very surprising; please explain."
The data can override this prior if the evidence is strong enough.
Otherwise the dials stay modest.
C PARAMETER: COUNTER-INTUITIVE DIRECTION
sklearn spells the leash strength as C, not lambda.
lambda is the fine-strength on squared dials; C is sklearn's name for its reciprocal.
Which means they are flipped versions of each other:
C = 1 / lambda <=> lambda = 1 / C
C = 0.1 -> lambda = 10 heavy penalty -- dials squeezed hard
C = 1.0 -> lambda = 1 sklearn default
C = 1000 -> lambda = 0.001 barely any penalty -- nearly free machine
check each on the slate: 1/0.1 = 10; 1/1.0 = 1; 1/1000 = 0.001
C=0.1 IS HEAVY PUNISHMENT, NOT LIGHT
C is the budget you give the machine to IGNORE the penalty.
A small budget means little room to ignore it -- a hard squeeze.
A large C means a large "ignore" budget -- so the penalty barely bites.
"C small = heavy leash" feels backwards until you remember C = 1/lambda.
DID THE DIALS SHRINK?
IN HAND so far: one S-curve machine (one dial per column, sum into z, squash z into a
chance).
Its leftover score is now two-part: plain fit (cross-entropy) plus a fine of lambda
times the sum of squared dials.
And the dictionary is C = 1/lambda, so the tight setting C = 0.1 means lambda = 1/0.1 =
10.
This section adds the receipt: proof the dials actually shrank.
Compare the average absolute dial size for the default leash (C=1.0) against the tighter
leash (C=0.1).
If the tighter leash is doing its job, its average dial comes out smaller.
(The two-line check is in the code at the end of the post.)
A concrete 3-column example, by pencil. Suppose only 3 columns -- x1, x2, x3 --
and two machines trained on the same data:
dial free (no leash) default C=1.0 tight C=0.1
------------------------------------------------------------
x1 dial +8.2 +3.1 +1.4
x2 dial -5.7 -2.4 -0.9
x3 dial +0.3 +0.2 +0.1
|dial| average:
free: (8.2 + 5.7 + 0.3) / 3 = 14.2 / 3 = 4.73
C=1.0: (3.1 + 2.4 + 0.2) / 3 = 5.7 / 3 = 1.90
C=0.1: (1.4 + 0.9 + 0.1) / 3 = 2.4 / 3 = 0.80
The free machine lets x1 balloon to 8.2 -- it bets heavily on one
column. C=1.0 pulls it to 3.1. C=0.1 pulls it to 1.4. The tighter
the leash, the more the machine spreads trust across all 3 columns.
Charge the FREE machine's dials (+8.2, -5.7, +0.3) the fine the tight leash
charges: lambda = 10 times the sum of squared dials. Work it on the slate
before reading on.
check your slate: 8.2^2 = 67.24; 5.7^2 = 32.49; 0.3^2 = 0.09; sum = 67.24 +
32.49 + 0.09 = 99.82; fine = 10 x 99.82 = 998.2. The tight machine's own dials
cost only 10 x (1.4^2 + 0.9^2 + 0.1^2) = 10 x (1.96 + 0.81 + 0.01) = 10 x 2.78 =
27.8. Ballooned dials cost about 36 times as much -- that is exactly the pressure
that makes the machine shrink them.
The absolute value (the number with its sign stripped off) is needed here.
Because a +3 dial and a -3 dial cancel in a plain average.
Which would make the machine look like it has no signal at all.
So we measure SIZE regardless of sign instead.
>> NOTE: WHY ABSOLUTE VALUE BEFORE AVERAGING
Dials can be positive or negative. A positive dial +3 and a negative dial -3 cancel to
zero in a plain average, making the machine look like it has no signal at all. Taking
the absolute value first measures the SIZE of each dial regardless of sign, then
averages those sizes. What you want to compare is pull strength, not direction.
SCALING AND THE LEASH ARE NOT THE SAME FIX
Scaling (put columns on one ruler): fixes the INPUTS
L2 leash: fixes the DIALS
Even after scaling, a dial can grow huge if the machine
over-trusts one column. You need both. They solve different problems.
Scaling means putting all 30 input columns on one ruler before the machine sees them.
The leash limits how large any individual dial grows while the machine sets its dials.
So removing either one leaves a different problem unsolved.
A COMPLETELY DIFFERENT MACHINE: THE TWO-CLOUD WALL
The problem this machine solves, drawn before the words:
two groups of dots in 2-column space (column A vs column B):
column B
| . . .
| . SICK . cloud of sick dots
| . . .
| |
| WELL | <- wall between the two clouds
| . . . . |
| . WELL . | cloud of well dots
| . . . . |
+----------------------------> column A
LDA finds the direction that MAXIMISES the gap between the two cloud centres
while MINIMISING how much the clouds spread in that direction.
Then it plants the wall perpendicular to that direction.
Score any new lump on that direction; which side it falls on is the label.
IN HAND so far: the leashed S-curve machine (fit plus lambda times squared dials, tight
setting C = 0.1, which means lambda = 1/0.1 = 10).
And the receipt that its dials shrank: average size (1.4 + 0.9 + 0.1)/3 = 2.4/3 = 0.80
against the default's (3.1 + 2.4 + 0.2)/3 = 5.7/3 = 1.90.
This section adds a second machine of the opposite temperament.
Everything so far kept the same machine and tightened its leash.
Now we change the machine itself.
The S-curve machine is a fidgeter -- it inches toward the answer by trial and error:
adjust, check the leftover, adjust again.
The new machine is LDA, which stands for linear discriminant analysis.
LDA has the opposite personality and does not fiddle at all.
It stands back, studies the SHAPE of the two groups of points, and lays down the wall
between them in a single confident stroke.
first split the study pile into sick rows and well rows
then compute 30 averages per group -> two centres in 30-column space
(60 averages total: 30 per class, 2 class centres)
then find the wall between the two centres
so a new lump -> which side of the wall? -> that is the label
well centre #----------+----------# sick centre
^
wall here
FISHER'S CRITERION: WHERE TO AIM THE WALL
A naive wall at the midpoint between the two centres works when both clouds are round.
But real clouds are stretched -- some directions have more spread than others.
And the two classes may share some of that stretch.
So a midpoint wall is not enough.
Fisher's criterion is the rule for aiming the wall well.
It asks: which direction w maximises the RATIO of between-class spread to within-class
spread?
Here w is the wall's NORMAL vector -- the arrow pointing straight across the wall, along
which we score each point.
maximise (w^T S_b w) / (w^T S_w w)
Here mu0 is the centre (average point) of class 0, and mu1 is the centre of class 1.
S_b is the between-class scatter: S_b = (mu1 - mu0)(mu1 - mu0)^T -- how far apart the two
centres sit.
S_w (written S_W) is the pooled within-class scatter: how spread-out the points are
inside each class, averaged across both.
So Fisher wants a direction where the centres are far apart but each cloud is tight.
The solution is closed-form -- no rolling downhill required:
w is proportional to S_W^-1 (mu1 - mu0)
Meaning: take the difference between the two class centres (mu1 - mu0).
Then rotate it by S_W^-1, the inverse of the pooled within-class spread matrix.
Which adjusts for the tilt and shape of the clouds.
So if both clouds are elongated diagonally, the wall tilts to match.
Where does the wall sit along that direction?
If the two classes are equally common, it sits exactly at the midpoint of the projected
class means:
threshold = w^T (mu0 + mu1) / 2 (only when the two classes are equally common)
A concrete 2-column, 4-person LDA walkthrough, by pencil.
Only 2 columns (bmi and bp) and 4 people:
person bmi bp truth
-----------------------------
A 0.04 0.90 sick (1)
B 0.06 0.85 sick (1)
C 0.12 0.50 well (0)
D 0.18 0.45 well (0)
First, the 60 averages (2 class means x 2 columns = 4 averages):
sick class (A, B): mu1 = ( (0.04+0.06)/2 , (0.90+0.85)/2 ) = (0.05 , 0.875)
well class (C, D): mu0 = ( (0.12+0.18)/2 , (0.50+0.45)/2 ) = (0.15 , 0.475)
Then the midpoint between the two centres (the IMAGINARY point):
midpoint = ((0.05+0.15)/2 , (0.875+0.475)/2 ) = (0.10 , 0.675)
Then the difference between centres:
mu1 - mu0 = (0.05-0.15 , 0.875-0.475) = (-0.10 , 0.400)
Then the wall NORMAL vector w (ignoring S_W for this clean round-cloud
picture; with equal covariances w = mu1 - mu0):
w = (-0.10 , 0.400)
Finally, project each person onto w and compare to the midpoint projection:
w^T midpoint = -0.10*0.10 + 0.400*0.675 = -0.01 + 0.270 = 0.260
A: w^T x = -0.10*0.04 + 0.400*0.90 = -0.004 + 0.360 = 0.356 > 0.260 -> sick
B: w^T x = -0.10*0.06 + 0.400*0.85 = -0.006 + 0.340 = 0.334 > 0.260 -> sick
C: w^T x = -0.10*0.12 + 0.400*0.50 = -0.012 + 0.200 = 0.188 < 0.260 -> well
D: w^T x = -0.10*0.18 + 0.400*0.45 = -0.018 + 0.180 = 0.162 < 0.260 -> well
All 4 classified correctly. The wall sits at 0.260 along w. New
lump with bmi=0.10, bp=0.70: w^T x = -0.01 + 0.28 = 0.270 > 0.260 -> sick.
The wall is the midpoint because both classes have equal counts here.
Same wall: w = (-0.10, 0.400), and a lump sits on the sick side when w^T x beats
0.260. A new lump walks in (made-up): bmi = 0.08, bp = 0.60. Score it.
check your slate: w^T x = -0.10 * 0.08 + 0.400 * 0.60 = -0.008 + 0.240 = 0.232.
0.232 < 0.260, so the lump falls on the WELL side of the wall -- called well.
But the two classes are usually NOT equally common.
So that pure midpoint is only a special case.
The full rule scores each new lump and adds a nudge for how common each class is:
score(x) = w^T x + w0,
w0 = -1/2 (mu0 + mu1)^T S_W^-1 (mu1 - mu0) + log(pi1 / pi0)
Here w0 is the offset -- the fixed number added to w^T x that decides where the wall
sits.
pi0 and pi1 are the priors: the share of the pile in class 0 and in class 1 (they add to
1).
The first piece of w0 is the midpoint term.
The extra log(pi1/pi0) term slides the wall toward the rarer class.
So the machine doesn't over-shout the rarer class.
THE WALL IS NOT EXACTLY HALFWAY ON IMBALANCED DATA
The Wisconsin sheet is roughly 63% well and 37% sick -- not equal.
So here pi0 (well) = 0.63 and pi1 (sick) = 0.37, which makes log(pi1/pi0) =
log(0.37/0.63), a negative number, not zero.
sklearn's LinearDiscriminantAnalysis() uses EMPIRICAL priors by default -- meaning it
reads pi0 and pi1 straight off the actual class counts.
So the boundary it fits carries that log(pi1/pi0) offset and sits OFF the midpoint.
If you place a wall at the pure midpoint and expect to reproduce sklearn's
predictions, you will be off.
To get the clean halfway wall, force equal priors:
LinearDiscriminantAnalysis(priors=[0.5, 0.5]).
GENERATIVE VS DISCRIMINATIVE
The S-curve machine and LDA arrive at the same final form -- a straight-line wall through
the 30-column space.
But they derive it through completely different reasoning.
P(sick|x) below means the chance a lump is sick given its columns x.
P(x|sick) means the reverse: the chance of seeing those columns x if the lump were known
to be sick.
The table:
Property S-curve machine Two-cloud wall (LDA)
------------- --------------------------- -----------------------------------
Approach models P(sick|x) directly models P(x|sick) and P(sick)
Solution iterative (roll downhill) closed form (one shot)
Assumption no distribution on x Gaussian columns, equal spread
Breaks when columns perfectly tangled spread assumption badly violated
Works better large pile, noisy columns small pile, Gaussian columns
LDA is a GENERATIVE model.
Generative means it imagines how each class makes its data: each class spits out points
in a Gaussian (bell-shaped) cloud.
Then it uses Bayes' theorem to flip P(x|sick) and P(sick) into P(sick|x).
The S-curve machine is the opposite: DISCRIMINATIVE, meaning it models P(sick|x) directly
and never imagines how the data was made.
Via that flip, LDA's P(sick|x) works out to sig(w^T x + w0).
sig is the sigmoid squash again, w is the wall normal, w0 is the offset.
So that is the same S-curve form as logistic regression.
Same wall shape, different route.
SAME ACCURACY: WHAT IT MEANS
On the Wisconsin breast cancer sheet, LDA and the S-curve machine give nearly identical
accuracy.
Two completely different approaches, same result.
That is not a coincidence.
AGREEMENT = CLEAN DATA, GENUINELY SEPARABLE CLASSES
When the two methods agree, the data is telling you the answer.
The S-curve machine sets dials to make the true bins as likely as it can.
LDA reads the two cloud shapes instead.
Both find the same dividing line because that line is clearly written in the data.
If the sheet were noisy or the two groups heavily overlapping, the two methods would
diverge.
And that disagreement would tell you something important: the boundary is ambiguous.
WHY SCALING ALSO MATTERS FOR LDA
LDA computes S_W, the pooled within-class spread matrix (how spread the points are inside
each class, averaged across both classes).
Suppose column "area" runs in the thousands while column "smoothness" runs in hundredths.
Then the area column dominates that spread matrix -- its large numbers swamp the entries.
Which distorts the wall-normal direction w.
So putting every column on the same ruler before feeding LDA keeps S_W balanced.
And that makes the direction w meaningful.
1. The free S-curve machine lets dials grow without limit.
L2 adds a squared-dial price that shrinks them toward zero.
2. C = 1/lambda, so small C (say 0.1) means large lambda (10) -- a heavy squeeze.
This direction feels backwards.
3. The leash fixes the dials; scaling fixes the inputs.
They solve different problems, so you need both.
4. LDA reads the two cloud centres (mu0, mu1) and their shared spread (S_W).
Then it finds the best wall in one closed-form shot.
5. w prop. S_W^-1 (mu1 - mu0): the difference of centres, rotated by the inverse
within-class spread.
6. LDA and the S-curve machine produce the same wall shape.
But they derive it from opposite directions -- generative vs discriminative.
7. When both machines agree on accuracy, the data is cleanly separable.
When they disagree, the boundary is ambiguous.
Nothing above needed a computer -- only pencils, clerks, and patience. This last
section is for the day you meet one: the same steps, spoken in Python.
First the dial comparison, L2 fine, and LDA projection from the worked examples:
3-COLUMN DIAL COMPARISON: FREE MACHINE VS C=1.0 VS TIGHT C=0.1
x1_free, x2_free, x3_free = +8.2, -5.7, +0.3
x1_def, x2_def, x3_def = +3.1, -2.4, +0.2
x1_tight,x2_tight,x3_tight = +1.4, -0.9, +0.1
avg_free = (abs(x1_free) +abs(x2_free) +abs(x3_free)) / 3 # (8.2+5.7+0.3)/3 = 4.73
avg_def = (abs(x1_def) +abs(x2_def) +abs(x3_def)) / 3 # (3.1+2.4+0.2)/3 = 1.90
avg_tight = (abs(x1_tight)+abs(x2_tight)+abs(x3_tight)) / 3 # (1.4+0.9+0.1)/3 = 0.80
print(round(avg_free,2), round(avg_def,2), round(avg_tight,2)) # 4.73 1.9 0.8
L2 FINE ON THE FREE MACHINE'S DIALS: LAMBDA=10 TIMES SUM OF SQUARED DIALS
lam = 10
fine_free = lam*(x1_free2 + x2_free2 + x3_free**2) # 10*(67.24+32.49+0.09) = 998.2
fine_tight = lam*(x1_tight2 + x2_tight2 + x3_tight**2) # 10*(1.96+0.81+0.01) = 27.8
print(round(fine_free,1), round(fine_tight,1)) # 998.2 27.8 (ballooned dials ~36x costlier)
C = 1/LAMBDA: SMALL C = HEAVY SQUEEZE
lam_tight = 1/0.1 # 10
lam_def = 1/1.0 # 1
lam_free = 1/1000 # 0.001
print(lam_tight, lam_def, lam_free) # 10.0 1.0 0.001
LDA 4-PERSON WALKTHROUGH: SICK A(0.04,0.90) B(0.06,0.85), WELL C(0.12,0.50) D(0.18,0.45)
mu1_bmi = (0.04 + 0.06) / 2 # 0.05
mu1_bp = (0.90 + 0.85) / 2 # 0.875
mu0_bmi = (0.12 + 0.18) / 2 # 0.15
mu0_bp = (0.50 + 0.45) / 2 # 0.475
print(mu1_bmi, mu1_bp, mu0_bmi, mu0_bp) # 0.05 0.875 0.15 0.475
w_bmi = mu1_bmi - mu0_bmi # -0.10 (wall normal)
w_bp = mu1_bp - mu0_bp # 0.400
threshold = w_bmi*(mu0_bmi+mu1_bmi)/2 + w_bp*(mu0_bp+mu1_bp)/2 # 0.260
print(round(threshold,3)) # 0.260
proj_A = w_bmi*0.04 + w_bp*0.90 # -0.004 + 0.360 = 0.356 > 0.260 -> sick
proj_B = w_bmi*0.06 + w_bp*0.85 # -0.006 + 0.340 = 0.334 > 0.260 -> sick
proj_C = w_bmi*0.12 + w_bp*0.50 # -0.012 + 0.200 = 0.188 < 0.260 -> well
proj_D = w_bmi*0.18 + w_bp*0.45 # -0.018 + 0.180 = 0.162 < 0.260 -> well
print(round(proj_A,3), round(proj_B,3), round(proj_C,3), round(proj_D,3))
0.356 0.334 0.188 0.162 -> ALL 4 CORRECT: A,B SICK, C,D WELL
All numbers from the worked examples, each on its own line. The toolbox block
below fits both machines on the full Wisconsin sheet:
Three small things: fit the tighter-leashed machine, check its dials really did shrink,
and fit the two-cloud wall (LDA) both ways -- empirical priors and forced-equal priors.
>> NEW TO PYTHON? Each named once:
np.abs(x) -- the size of each number, sign thrown away (NumPy)
np.mean(x) -- the average of a row of numbers
Tighter leash (C=0.1 is a HEAVY squeeze -- remember C = 1/lambda):
log_reg_l2 = LogisticRegression(penalty='l2', C=0.1, random_state=42)
log_reg_l2.fit(X_train_scaled, y_train)
y_pred_l2 = log_reg_l2.predict(X_test_scaled)
Did the dials shrink? Compare the average absolute dial size, default vs tighter:
avg_coef_baseline = np.mean(np.abs(log_reg_baseline.coef_)) # C=1.0 (default)
avg_coef_l2 = np.mean(np.abs(log_reg_l2.coef_)) # C=0.1 (tighter)
EXPECT: AVG_COEF_L2 < AVG_COEF_BASELINE (THE TIGHTER LEASH SHRINKS THEM FURTHER)
The two-cloud wall, both ways -- sklearn's default uses empirical priors, so force equal
priors if you want the exact halfway wall:
SKLEARN DEFAULT: EMPIRICAL PRIORS -> WALL SHIFTED OFF THE MIDPOINT
lda = LinearDiscriminantAnalysis()
lda.fit(X_train_scaled, y_train)
y_pred_lda = lda.predict(X_test_scaled)
TO REPRODUCE THE EXACT "HALFWAY BETWEEN THE CENTRES" WALL:
lda_equal = LinearDiscriminantAnalysis(priors=[0.5, 0.5])
lda_equal.fit(X_train_scaled, y_train)
Plain term used above Standard label
----------------------------------- ------------------------------------------
leash on the dials L2 regularisation / ridge penalty
dial-size price regularisation term lambda*sum(beta_j^2)
C (sklearn parameter) inverse regularisation strength (C = 1/lambda)
two-cloud midpoint wall LDA (linear discriminant analysis)
cloud centre class mean mu_k
pooled within-class spread within-class scatter matrix S_W
Fisher's criterion maximise (w^T S_b w)/(w^T S_w w)
models P(x|class) generative model
models P(class|x) directly discriminative model
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 3 -- Sorting Into Bins):
Part 1 -- The S-Curve, the Four-Box Table .
Part 2 -- The Trade Curve .
Part 3 (this post) .
Part 4 -- Picking Settings, Skewed Piles
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================