==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
APPENDIX . CLASSIFICATION REFERENCE
Loss, Leash, Grid, and All the Terms
============================================================================================
This is a flip-to reference on sorting things into bins.
Sorting into bins means a yes/no machine.
It reads a row of numbers and shouts one of two labels, like "sick" or "well".
Unlike a walk-through lesson, this page sits idea and code side by side for quick lookup.
It gathers the key classification ideas into ten tight sections.
This page assumes you remember nothing from anywhere else.
Every term is built from scratch where it appears.
Plain language first, standard labels at the very bottom.
1. WHY CROSS-ENTROPY, NOT MSE
Linear regression is the straight-stick rule: it fits a line to predict a sliding number.
It scores itself by squared leftovers.
A leftover is guess minus truth.
If the guess is 2.3 and the truth is 2.0, the leftover squared is (2.3-2.0)^2 = 0.3^2 = 0.09.
That ruler is fine for sliding numbers.
But a bin-sorter is different: its truth is 0 or 1, and its guess is a chance between 0 and 1.
So squared distance is the wrong ruler here.
It punishes a wrong answer by the same amount whether the machine was nearly right or bone-backwards.
Cross-entropy (log-loss) punishes backwards confidence exponentially:
L = -(1/n) sum [ y * log(p) + (1-y) * log(1-p) ]
where p is the machine's chance output and y is the true label (0 or 1).
truly sick (y=1), machine says p=0.99 -> -log(0.99) ~= 0.01 tiny
truly sick (y=1), machine says p=0.01 -> -log(0.01) ~= 4.6 huge
truly well (y=0), machine says p=0.01 -> -log(0.99) ~= 0.01 tiny
truly well (y=0), machine says p=0.99 -> -log(0.01) ~= 4.6 huge
work the big one on the slate: log(0.01) = -log(100), and log(100) = 2 x log(10)
~= 2 x 2.303 = 4.606, so -log(0.01) ~= 4.6. The near-miss: -log(0.99) ~= 0.01,
since shaving 1% off 1 barely moves the log.
A truly-sick lump (y = 1) is handed chance p = 0.5 by a hedging machine. Work its
cross-entropy fine.
check your slate: fine = -log(p) = -log(0.5) = log(2) ~= 0.693. A coin-flip
hedge costs about 0.69 -- more than the confident-right 0.01, far less than the
confident-wrong 4.6. Hedging is punished gently; being sure and wrong is not.
The machine is penalised hardest for being CONFIDENT and WRONG.
MSE means mean squared error: average the squared leftovers.
MSE would score that last case (truly well, machine says 0.99) as (0.99-0)^2 = 0.99 x 0.99 = 0.9801, call it 0.98.
That is large but bounded -- it can never grow past 1.
Log-loss drives toward infinity instead.
Which means the gradient (the downhill slope the machine slides on) always points away from confident wrong answers.
USE LOG-LOSS FOR CLASSIFICATION, MSE FOR REGRESSION
sklearn's LogisticRegression minimises log-loss by default. MSE has no ceiling on a
chance output in [0,1], so it produces badly calibrated probabilities.
2. THE LEASH: L2 REGULARISATION AND THE C PARAMETER
sklearn is the standard Python machine-learning library.
LogisticRegression is its yes/no machine; its dials are the numbers it tunes to fit the data.
The default LogisticRegression() in sklearn is NOT a free machine.
It carries a leash built in:
penalty='l2', C=1.0 (the sklearn default)
The leash adds a term to the loss that punishes large dials.
log-loss is the classification score from section 1: it drives confident-wrong guesses toward infinity.
beta_j is one dial of the machine; sum(beta_j^2) adds up the squares of all the dials.
total cost = log-loss + (1/C) * sum(beta_j^2)
C is the inverse of the leash tightness.
Small C means a tight leash, which means dials squeezed toward zero.
Large C means a loose leash, which means dials can grow freely.
C = 0.01 very tight -- dials squeezed hard, machine forced simple
C = 0.1 tight
C = 1.0 medium (sklearn default)
C = 10 loose
C = 1000 nearly free -- almost no squeeze
A machine has just two dials, 2 and 3, under a tight leash C = 0.1 (made-up).
Work the leash's slice of the cost, (1/C) * sum(beta^2).
check your slate: 1/C = 1/0.1 = 10; sum of squares = 2^2 + 3^2 = 4 + 9 = 13;
leash cost = 10 x 13 = 130. Loosen the leash to C = 10 and the same dials cost
only (1/10) x 13 = 1.3 -- the tight leash makes big dials hurt a hundred times more.
C IS THE INVERSE OF LAMBDA
In textbooks regularisation strength is written as lambda, and the leash term is
lambda * sum(beta^2). sklearn inverts it: C = 1/lambda. More C means LESS squeeze.
Easy to flip the direction when tuning.
To remove the leash entirely:
LogisticRegression(penalty=None)
L2 is the squeeze named above: the penalty that adds (1/C) * sum(beta^2) to the cost.
Adding L2 helps when the training pile is small relative to the number of columns.
It also helps when columns are correlated.
Correlated columns let dials swing wildly without a check, and the squeeze stops that.
3. A SECOND SORTER: LINEAR DISCRIMINANT ANALYSIS
Logistic regression learns its boundary by gradient descent on log-loss.
Gradient descent means rolling the dials downhill on the cost a little at a time.
LDA stands for Linear Discriminant Analysis, and it takes a different road.
It assumes each class is a cloud of points drawn from a Gaussian.
A Gaussian is a bell-shaped scatter; its shape is its covariance, the matrix of spreads and tilts.
LDA assumes both clouds share one shape.
Then it computes the mean centre of each cloud.
So it places the boundary where the two clouds are equally likely to have produced a new point.
logistic regression -- learns boundary from data; no cloud-shape assumption
LDA -- assumes Gaussian clouds, equal shape; boundary from cloud means
The boundary LDA draws is a LINEAR wall -- a straight divider, the same kind logistic regression draws.
But LDA computes it analytically (by a direct formula), not by rolling downhill.
The wall normal direction (the direction the wall faces) is:
w = S_W^-1 * (mu1 - mu0)
S_W is the within-class scatter: the two clouds' shared spread-and-tilt, pooled into one matrix.
S_W^-1 is its inverse, the matrix that un-stretches that spread.
mu0 and mu1 are the two class mean vectors: the centre point of class 0 and the centre point of class 1.
The wall is placed at:
w0 = -1/2 * (mu0+mu1)^T * S_W^-1 * (mu1-mu0) + log(pi1/pi0)
pi0 and pi1 are the priors: the share of all rows that are class 0 and the share that are class 1.
log(pi1/pi0) is the LOG-PRIOR term.
If the sick pile is smaller, then pi1 < pi0, so this term is negative.
Which means the wall shifts toward the sick cloud.
So the machine is already sceptical about sick cases.
>> NOTE: EQUAL PRIORS PUTS THE WALL AT THE MIDPOINT
The priors pi0, pi1 are the shares of each class.
If pi0 = pi1 = 0.5, then pi1/pi0 = 1 and log(1) = 0, so the log-prior term vanishes.
Which means the wall sits exactly halfway between the two cloud centres.
sklearn's default uses EMPIRICAL priors instead -- the class frequencies counted in the training data.
A common breast-tumour dataset (the Wisconsin set) is about 63% benign.
So the priors are unequal, and the wall shifts off the midpoint.
lda = LinearDiscriminantAnalysis() # empirical priors, shifted wall
lda_mid = LinearDiscriminantAnalysis(priors=[0.5,0.5]) # equal priors, midpoint wall
4. SETTINGS VS DIALS: HYPERPARAMETERS
Every machine has two kinds of knobs:
dials (parameters) -- set BY THE MACHINE during training to fit the data
e.g. beta_0 ... beta_30 in logistic regression
settings (hyperparams) -- set BY YOU before training; the machine never touches them
e.g. C in LogisticRegression, n_neighbors in KNN
A setting controls HOW the machine learns, not WHAT it learns.
You cannot find the best setting by watching the training pile.
The machine can always overfit if you give it enough slack.
Overfit means memorise the training rows instead of learning the real pattern.
So you find the best setting by watching a HELD-OUT pile -- rows kept aside, called a validation fold.
dials -> machine finds them (by gradient descent, or analytically)
settings -> YOU find them (by grid search over a validation set)
5. THE GRID HUNT: FINDING THE BEST SETTING
You want the best C, the leash setting from section 2 -- small C squeezes the dials, large C frees them.
Candidates to try: [0.01, 0.1, 1, 10].
Do NOT just try each on the training pile.
The machine memorised that pile, so it will look better the looser the leash.
Instead ROTATE the data.
Split the training pile into k equal folds (k=5 is common).
Train on k-1 of the folds, then score on the one left-out fold.
Rotate so a different fold is left out, and repeat k times.
So you average the k scores into one number per candidate.
first pick a range of candidate settings
then for each candidate, score it with k-fold cross-validation
so pick the candidate with the best mean score
This is grid search. "Grid" because you can tune multiple settings at once -- a 2-D grid
of C values crossed with penalty types, for example.
SCALE INSIDE THE FOLD, NOT BEFORE
If you StandardScale the whole training pile first, then pass it to GridSearchCV, each
validation fold was shaped by a scaler that already saw it. The mean and spread used to
scale the held-out data leaked out of it. Put the scaler INSIDE a Pipeline:
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
pipe = Pipeline([
('scaler', StandardScaler()),
('clf', LogisticRegression()),
])
param_grid = {'clf__C': [0.01, 0.1, 1, 10]}
gs = GridSearchCV(pipe, param_grid, cv=5, scoring='recall')
gs.fit(X_train, y_train) # pass RAW X_train here, not pre-scaled
Three things you must specify.
First, the range and spacing of candidates.
Then, how many folds to rotate (k).
And, which score to optimise -- recall, F1, or AUC, all defined in the sections below.
6. PINCH-TO-FIT: MIN-MAX SCALING
StandardScaler shifts each column to mean 0, spread 1.
Mean 0 means subtract the average; spread 1 means divide by the standard deviation.
Min-max scaling squeezes each column into the range [0, 1] instead:
x_scaled = (x - min(x)) / (max(x) - min(x))
min(x) is the smallest value in the column; max(x) is the largest.
Every column is "pinched" to fit between 0 and 1.
This keeps the distribution shape but compresses the range.
raw: [ 100, 200, 400, 800 ]
min = 100, max = 800, range = max - min = 800 - 100 = 700
(100-100)/700 = 0/700 = 0.00
(200-100)/700 = 100/700 = 0.14
(400-100)/700 = 300/700 = 0.43
(800-100)/700 = 700/700 = 1.00
scaled: [ 0.0, 0.14, 0.43, 1.0 ]
ONE OUTLIER SQUASHES EVERYTHING ELSE
If a column has one value of 10000 and everything else sits between 1 and 100, the
min-max range is ~9999. After scaling, the bulk of the data squeezes into 0 to 0.01 --
a thin sliver. Standard scaling handles outliers better.
use STANDARD (mean 0, spread 1) when: roughly bell-shaped, no extreme outliers
use MIN-MAX (0 to 1) when: bounded range required, known clean limits,
or feeding a neural network / image model
7. THE LIVES-VS-MONEY TRADE: PRECISION AND RECALL IN BUSINESS
CAUGHT means a sick case the machine correctly shouted sick.
MISSED means a sick case the machine wrongly called well.
ALARM means a well case the machine wrongly shouted sick.
Precision and recall pull in opposite directions.
The business situation decides which to favour.
recall = CAUGHT / (CAUGHT + MISSED) -- share of truly sick cases found
precision = CAUGHT / (CAUGHT + ALARM) -- share of sick shouts that were real
scenario A: cancer screening
MISSED = cancer sent home untreated = catastrophic
ALARM = extra biopsy = costly but survivable
-> maximise RECALL, tolerate lower precision
scenario B: spam filter
MISSED = spam in inbox = annoying
ALARM = good email blocked = catastrophic (missed invoice, job offer)
-> maximise PRECISION, tolerate spam slipping through
scenario C: fraud detection
MISSED = fraud unblocked = costly to the bank
ALARM = good transaction blocked = customer complaint
-> tune recall first, set a floor on precision
The cutoff is the chance above which the machine shouts sick.
Raising the cutoff means more sure before shouting sick, which raises precision and drops recall.
Lowering the cutoff raises recall and drops precision.
F1 is the harmonic mean of precision and recall: F1 = 2 * precision * recall / (precision + recall).
Which means F1 collapses toward zero if EITHER one is near zero.
>> NOTE: USE F-BETA TO TILT THE TRADE
F1 weights precision and recall equally. F-beta with beta > 1 weights recall more
heavily. F2 (beta=2) counts a miss twice as costly as a false alarm.
8. THE TRADE CURVE REVISITED: WHEN AUC MISLEADS
The trade curve, standard name ROC curve, sweeps the cutoff from 1 down to 0.
The cutoff is the chance above which the machine shouts sick.
At each cutoff it plots a point (FPR, TPR).
TPR is the true-positive rate = CAUGHT / all truly sick (the recall).
FPR is the false-positive rate = ALARM / all truly well, where ALARM is a well case wrongly shouted sick.
AUC is the area under that curve.
AUC = 1.0 means perfect; AUC = 0.5 means coin flip.
AUC does not depend on where you set the cutoff.
It answers one question: how cleanly do the two groups separate?
So it is the right score to COMPARE two machines before deciding where to set the cutoff.
But AUC has a blind spot.
Its x-axis is FPR = ALARM / all truly well.
When the well pile is huge -- say a rare-disease screen with 1 sick per 100 well -- that denominator is large.
So FPR stays small even when there are many alarms.
Which means the ROC curve looks optimistic.
Precision = CAUGHT / (CAUGHT + ALARM) tells a different story: most "sick" shouts are wrong.
For SKEWED PILES, the Precision-Recall curve tells the truth:
x-axis = RECALL (how many sick cases found)
y-axis = PRECISION (of the sick shouts, how many were real)
A true negative is a well case correctly called well.
The PR curve ignores the true-negative count entirely.
So it cannot be flattered by a large well pile.
High area under the PR curve means the machine finds sick cases AND its sick shouts are reliable.
USE ROC/AUC FOR BALANCED CLASSES; USE PR CURVE FOR SKEWED CLASSES
A machine with AUC 0.95 can have precision 0.10 on a 1:100 sick-to-well pile.
PR curves surface this; ROC curves hide it.
9. SKEWED PILES: WHAT GOES WRONG AND HOW TO FIX IT
Skewed classes, also called imbalanced classes, means one label is far rarer than the other.
This is the norm in real classification tasks.
Fraud is under 1% positive; rare disease is a few % positive; churn is 10-20% positive.
Accuracy means the share of all rows the machine labelled correctly.
When the sick pile is tiny, accuracy flatters the lazy machine:
pile: 95 well, 5 sick
machine: shout well for everything
accuracy = 95/100 = 0.95 <- looks great
recall = 0/5 = 0.00 <- catches nobody
precision = N/A <- never shouted sick
Fixes to try when the pile is skewed:
1. report recall and precision instead of accuracy
2. use the PR curve instead of the ROC curve
3. tune the cutoff: lower it to increase recall at the cost of precision
4. oversample the rare class (SMOTE), undersample the common class, or
use class_weight='balanced' in sklearn to upweight the rare class
LogisticRegression(class_weight='balanced')
-> internally scales the log-loss contribution of each class by
n_samples / (n_classes * n_samples_per_class)
The machine sees each rare-class mistake as proportionally more costly.
So it stops defaulting to the common class.
10. COUNTING ACROSS CLASSES: MACRO, MICRO, WEIGHTED
When there are more than two bins -- say tumour types A, B, C -- you get one precision and one recall per class.
Precision is the share of a class's shouts that were right; recall is the share of a class's true members found.
Three ways collapse those per-class numbers into a single number:
MACRO: compute the metric per class, average with EQUAL WEIGHT
-> every class counts the same, rare and common alike
MICRO: pool all CAUGHT, ALARM, MISSED across classes first, THEN compute
-> dominated by the biggest class; equals accuracy for precision/recall/F1
WEIGHTED: average the per-class metrics, weighted by each class's count
-> common classes count more, rare classes less
Example with three classes, sizes 80, 15, 5:
class A class B class C
recall: 0.90 0.60 0.30
count: 80 15 5
macro = (0.90 + 0.60 + 0.30) / 3 = 1.80 / 3 = 0.60
weighted: 0.90*80 = 72 ; 0.60*15 = 9 ; 0.30*5 = 1.5 ; sum = 82.5 ; /100 = 0.82
micro = (all TP) / (all TP + all FN) ~= 0.84 (dominated by class A)
TP means true positives (members found); FN means false negatives (members missed).
macro treats a 5-sample class the same as an 80-sample class.
Use macro when all classes matter equally.
Use weighted when you care more about getting the big classes right.
Use micro when total correct counts are what matters.
classification_report DOES NOT PRINT "micro avg"
For single-label classification, micro precision/recall/F1 all equal accuracy, so
sklearn prints "accuracy" instead of a "micro avg" row. To get micro explicitly:
from sklearn.metrics import precision_recall_fscore_support
micro = precision_recall_fscore_support(y_test, y_pred, average='micro')
macro = precision_recall_fscore_support(y_test, y_pred, average='macro')
Plain term used above Standard label
------------------------------------ -------------------------------------------
cross-entropy leftover binary cross-entropy / log-loss
dial squeeze L2 regularisation / ridge penalty
leash tightness (inverse) C (regularisation parameter in sklearn)
lambda regularisation strength (C = 1/lambda)
linear separator from cloud means Linear Discriminant Analysis (LDA)
log(pi1/pi0) log-prior ratio / class-balance offset
within-class scatter S_W (pooled within-class covariance matrix)
setting not learned by the machine hyperparameter
dial learned by the machine parameter / coefficient / weight
grid hunt over validation folds grid search + cross-validation / GridSearchCV
rotating folds k-fold cross-validation (cv=k)
pinch-to-fit scaling min-max normalisation / MinMaxScaler
recall matters more high-recall regime; use F-beta (beta > 1)
precision matters more high-precision regime; use F-beta (beta < 1)
curve of precision vs recall Precision-Recall (PR) curve
area under PR curve AUCPR / average precision score
equal weight per class macro average
count-weighted per class weighted average
pool all counts first micro average
upweight the rare class class_weight='balanced'
----------------------------------------------------------------------------------------------
SEE ALSO (Chapter 3 -- Sorting Into Bins):
Part 1 -- The S-Curve, the Four-Box Table .
Part 2 -- The Trade Curve .
Part 3 -- Leash and Cloud .
Part 4 -- Picking Settings, Skewed Piles
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================