==============================================================================================
RAHUL'S ML BLOG -- notes on machine learning, worked out by hand est. 2026
==============================================================================================
home | about | archive | glossary | contact
----------------------------------------------------------------------------------------------
CHAPTER 3 . SORTING INTO BINS . PART 2 OF 4
The Trade Curve: Sliding the Cutoff and What AUC Actually Measures
============================================================================================
Here is the setup, from scratch.
A machine reads a lump from a breast scan.
The machine puts out a chance-score: a number from 0 to 1.
It builds that number by adding up weighted dials, then squashing the sum into the 0-to-1
range with an S-shaped squash.
0 means "almost surely well".
1 means "almost surely sick".
To turn that score into a yes/no call you draw a line on the 0-to-1 scale.
That line is the cutoff (also called the threshold).
Above the line the machine says "sick"; below it the machine says "well".
A common habit is to draw the line at 0.5.
Several common scores -- the ones that count right and wrong calls -- assume that 0.5 line.
But who said 0.5?
It is just one opinion about where "probably well" ends and "probably sick" begins.
Therefore it is an opinion you are free to overrule.
Reach out and slide that line.
Push it left and the machine turns anxious, crying "sick" at the faintest shadow.
Which means it catches more real cases.
But that startles more healthy people with false alarms.
Push it right and the machine turns stoic, holding its tongue until it is almost certain.
Which means fewer false alarms.
But that lets more real cases slip quietly out the door.
So nowhere on that line is there a free lunch.
Every spot you pick is a different bargain struck between lives and money.
The trade curve is what you get when you stop arguing over a single spot.
It lays all the bargains out side by side, in one picture.
Machine A hands each lump a score. The two sick lumps get 0.49 and 0.48; the two
well lumps get 0.01 and 0.02. Grade it at the usual cutoff of 0.5:
0.0 0.01 0.02 0.48 0.49 1.0
|------oo----------------------------ss--|--------
well ^ cutoff 0.5
sick scores 0.48, 0.49 sit BELOW 0.5 -> both called "well"
At 0.5 the machine catches nobody and looks worthless. But look again -- it put
both sick lumps ABOVE both well lumps. The separation is perfect; the cutoff is
just standing in the wrong spot. Slide it down to 0.40:
0.0 0.01 0.02 0.40 0.48 0.49 1.0
|------oo----------------|-----------ss----------
cutoff here -> both sick caught, zero false alarms
So a single cutoff can bury a perfect machine. Never grade on one line. Sweep the
cutoff from 1 down to 0 and trace the catches and false alarms the whole way --
that sweep is the trade curve, the ROC curve.
IN HAND: the machine gives each lump a chance-score from 0 to 1.
You draw a line (the cutoff) somewhere on that scale.
Above the line the machine calls "sick"; below the line it calls "well".
Every lump now lands in one of four boxes, set by truth-vs-call.
CAUGHT (truly sick, called sick) -- the standard name is TP, true positive.
FALSE ALARM (truly well, called sick) -- the standard name is FP, false positive.
MISSED (truly sick, called well) -- the standard name is FN, false negative.
CLEARED (truly well, called well) -- the standard name is TN, true negative.
This section measures the two rates those boxes produce as the cutoff slides.
TWO RATES THE BOXES FORCE OUT
At any fixed cutoff, the four boxes yield two fractions.
The first is the catch-rate, written TPR (true positive rate).
It is the share of truly sick lumps the machine caught.
catch-rate (TPR) = CAUGHT / all truly sick ^ good
The second is the false-alarm rate, written FPR (false positive rate).
It is the share of truly well lumps the machine wrongly called sick.
false-alarm (FPR) = ALARM / all truly well v good
Written out with symbols: Y is the true verdict (1 sick, 0 well).
yhat is the machine's called verdict.
t is the cutoff.
TP, FP, FN, TN are the counts in the four boxes (CAUGHT, ALARM, MISSED, CLEAR).
TPR(t) = P(yhat=1 | Y=1) = TP(t) / (TP(t) + FN(t))
FPR(t) = P(yhat=1 | Y=0) = FP(t) / (FP(t) + TN(t))
TPR is the same thing as recall: caught sick over all truly sick.
FPR is the fraction of healthy lumps that get misread as sick.
Moving the cutoff shifts both at once.
Therefore they cannot be improved together without limit.
SLIDING THE CUTOFF
chance 0.0 --------------- 1.0
^ cutoff (slide)
slide LEFT (cutoff ~= 0.1): shout sick at almost everyone
catch-rate ^ (few sick lumps escape)
false-alarm ^ (healthy people caught in the net too)
slide RIGHT (cutoff ~= 0.9): shout sick only when bone-sure
catch-rate v (many sick lumps called well and sent home)
false-alarm v (healthy people mostly called clear)
-> a TRADE: cannot raise catches without raising false alarms
Recall TPR is the catch-rate (caught sick over all truly sick) and FPR is the false-alarm
rate (well lumps wrongly called sick over all truly well).
There is no cutoff that maximises TPR while holding FPR at zero.
The one exception is a machine that perfectly separates the two groups.
So what exists instead is a continuous family of deals.
DRAWING ALL THE DEALS AT ONCE
Sweep the cutoff from 1 down to 0.
At each cutoff position, compute the false-alarm rate FPR and the catch-rate TPR.
FPR is well lumps wrongly called sick over all truly well.
TPR is sick lumps caught over all truly sick.
Drop a dot at the coordinates (FPR, TPR).
Connect the dots.
That joined line is the trade curve, also called the ROC curve.
TPR (catch-rate, lives saved)
1 | * <- perfect machine (catch all, alarm none)
| ###
| ##
| - - - - \ <- diagonal = useless coin flip
0 +--------------- FPR (false-alarm, money wasted)
0 1
each dot = one cutoff position
perfect machine: hugs the top-left corner
coin flip: follows the diagonal
The curve is the set of dots traced as the cutoff t runs over 0 to 1: {(FPR(t), TPR(t))}.
As t -> 0, the line sits at the bottom, so everyone scores above it and is called sick.
Which means every sick lump is caught (TPR=1) and every well lump is alarmed (FPR=1).
That puts the dot at the top-right corner.
As t -> 1, the line sits at the top, so everyone scores below it and is called well.
Which means no sick lump is caught (TPR=0) and no well lump is alarmed (FPR=0).
That puts the dot at the bottom-left corner.
A concrete 6-person exam pile, by pencil:
exam pile: 3 sick (truth=1), 3 well (truth=0)
lump truth machine chance 6 cutoff positions
-------------------------------------------------------------
A sick 0.97 -> cutoff 0.99: below -> well FN
B sick 0.88 -> cutoff 0.90: below -> well FN
C sick 0.72 -> cutoff 0.80: below -> well FN
D well 0.45 -> cutoff 0.60: below -> well TN
E well 0.22 -> cutoff 0.40: below -> well TN
F well 0.11 -> cutoff 0.20: below -> well TN
Sweep the cutoff through 6 positions and compute (FPR, TPR):
cutoff >=cut? CAUGHT MISSED ALARM CLEAR TPR FPR
-----------------------------------------------------------------
0.99 none 0/3 3/3 0/3 3/3 0.000 0.000
0.90 A 1/3 2/3 0/3 3/3 0.333 0.000
0.80 A,B 2/3 1/3 0/3 3/3 0.667 0.000
0.60 A,B,C 3/3 0/3 0/3 3/3 1.000 0.000
0.40 A,B,C,D 3/3 0/3 1/3 2/3 1.000 0.333
0.20 A,B,C,D, 3/3 0/3 2/3 1/3 1.000 0.667
E
0.00 all 3/3 0/3 3/3 0/3 1.000 1.000
TPR = CAUGHT / (CAUGHT + MISSED). FPR = ALARM / (ALARM + CLEAR).
At cutoff 0.90: 1 sick caught, 2 missed -> TPR=1/3=0.333;
0 alarms -> FPR=0/3=0.000. At cutoff 0.40: all 3 sick caught +
1 well alarmed -> TPR=3/3=1.000, FPR=1/3=0.333.
Plot the 6 dots: (0.000,0.000), (0.000,0.333), (0.000,0.667),
(0.000,1.000), (0.333,1.000), (0.667,1.000), (1.000,1.000).
The curve hugs the left edge (FPR=0 for the first 4 cutoffs),
then bends right. The curve is ABOVE the diagonal -- the machine
is doing real work.
At some cutoff (made-up) the four-box holds, out of 4 sick and 6 well: CAUGHT 3,
MISSED 1, ALARM 2, CLEAR 4. Work the catch-rate and false-alarm rate.
check your slate: TPR = CAUGHT / (CAUGHT + MISSED) = 3 / (3 + 1) = 3/4 = 0.75;
FPR = ALARM / (ALARM + CLEAR) = 2 / (2 + 4) = 2/6 ~= 0.333. One dot on the
trade curve at (0.333, 0.75) -- three sick in four caught, one well in three
falsely alarmed.
Count the work of one sweep.
With n lumps you get n + 1 cutoff positions (one above the top score, one below each score).
At each position you re-tally the four boxes over all n lumps.
So the total comparisons are (n + 1) x n.
The Wisconsin breast-scan exam has n = 569 lumps.
Therefore (569 + 1) x 569 = 570 x 569 = 324,330 comparisons to draw the whole curve.
That is a morning for the clerks, an afternoon's nap for the kettle.
AREA UNDER THE CURVE
IN HAND: a 6-person exam swept through cutoffs into (FPR, TPR) dots.
FPR is the false-alarm rate (well lumps wrongly called sick over all truly well).
TPR is the catch-rate (sick lumps caught over all truly sick).
Those dots climbed the left edge -- (0,0.333), (0,0.667), (0,1.0) -- then bent right.
This section crushes that whole staircase into one number.
The whole curve collapses into one number: the area under it, called AUC (area under curve).
A perfect machine hugs the top-left corner, so its area is 1.0.
A coin flip follows the diagonal, so its area is 0.5.
A machine whose scores run backwards drops below the diagonal, so its area is below 0.5.
AUC = 1.0 perfect -- no overlap between sick and well scores
AUC = 0.5 useless -- sick and well scores interleaved randomly
AUC < 0.5 backwards -- sick scores lower than well scores
AUC is computed from the curve dots by the trapezoidal rule.
The trapezoidal rule means: between two neighbouring dots, the strip of area is a trapezoid.
Its area is its width times its average height: (FPR gap) x (the two TPR heights averaged).
Add up every strip and you have the whole area under the curve.
Two neighbouring ROC dots (made-up): (FPR 0.2, TPR 0.6) then (FPR 0.5, TPR 0.9).
Work the area of the trapezoid strip between them: width x average height.
check your slate: width = 0.5 - 0.2 = 0.3; average height = (0.6 + 0.9)/2 =
1.5/2 = 0.75; strip = 0.3 x 0.75 = 0.225. Add up every such strip across the
sweep and you have the whole area under the curve.
Now do that for the real 6-cutoff sweep above, by pencil.
Each row takes one FPR-gap times the average of the two TPR heights:
drop FPR TPR (FPR_next - FPR) x (TPR_next + TPR) / 2
----------------------------------------------------------------------
1 0.000 0.000 (0.000 - 0.000) x (0.333 + 0.000) / 2 = 0.000
2 0.000 0.333 (0.000 - 0.000) x (0.667 + 0.333) / 2 = 0.000
3 0.000 0.667 (0.000 - 0.000) x (1.000 + 0.667) / 2 = 0.000
4 0.000 1.000 (0.333 - 0.000) x (1.000 + 1.000) / 2 = 0.333
5 0.333 1.000 (0.667 - 0.333) x (1.000 + 1.000) / 2 = 0.333
6 0.667 1.000 (1.000 - 0.667) x (1.000 + 1.000) / 2 = 0.333
--
total AUC = 0.000 + 0.000 + 0.000 + 0.333 + 0.333 + 0.333 = 1.000
The AUC is 1.0, which means a perfect ranking.
Indeed the machine sorted all 3 sick chances (0.97, 0.88, 0.72) above all 3 well chances
(0.45, 0.22, 0.11).
So for ANY cutoff between 0.72 and 0.45, the machine catches all sick and alarms none.
WHAT AUC ACTUALLY MEASURES
Areas under curves are easy to compute and hard to feel.
So here is the same number told as a tiny story you can picture.
This story goes by the name pick-higher reading.
Here x+ means a sick lump and x- means a well lump; score(.) is the machine's chance-score:
AUC = P( score(x+) > score(x-) )
Pick one sick lump at random and one well lump at random.
Ask: what is the chance the machine gave the sick lump a higher chance-score than the well
lump?
That chance is the AUC.
GOOD machine -- scores:
well lumps -> #### (all low)
sick lumps -> #### (all high)
-> almost always: sick score > well score -> AUC ~= 1.0
BAD machine -- scores:
well lumps -> # # # # # #
sick lumps -> # # # # # #
-> roughly half the time sick scores below well -> AUC ~= 0.5
Read AUC = 0.97 like this.
Take a random sick lump and a random well lump.
97 times out of 100 the machine scored the sick lump higher than the well lump.
So AUC answers "how cleanly do the two groups separate?"
And it answers without committing to any one cutoff.
AUC IS CUTOFF-INDEPENDENT
Recall the four box-counts at a fixed cutoff: CAUGHT (truly sick, called sick),
ALARM (truly well, called sick), MISSED (truly sick, called well), CLEAR (truly well,
called well).
Four common scores are built from those counts, and each one assumes a single fixed
cutoff (the usual 0.5):
accuracy = (CAUGHT + CLEAR) / everyone -- share of all calls that were right
recall = CAUGHT / (CAUGHT + MISSED) -- share of truly sick that were caught
precision = CAUGHT / (CAUGHT + ALARM) -- share of sick-calls that were right
F1 = 2 x precision x recall / (precision + recall) -- the blend of the two
All four answer "how good is this machine at one chosen cutoff?"
But that pins you to one cutoff.
AUC instead answers "how good is this machine's RANKING of sick chances above well
chances, everywhere, at every cutoff at once?"
So it is the right score when you want to compare two machines before deciding where
to set the cutoff.
WHY YOU NEED CHANCES, NOT LABELS
The ROC sweep needs a continuous score to slide the cutoff through.
predict is a function that returns only the final 0/1 call per row.
Two unique values means two dots, which means a useless curve.
predict_proba is a function that returns the raw chance-score (the 0-to-1 number) before
any cutoff is applied.
That gives a different number for every row.
So the sweep has something to slide through.
predict(X_test_scaled) -> [1, 0, 1, 1, 0, ...] (0 or 1 only)
predict_proba(X_test_scaled) -> [[0.03, 0.97], (well-chance, sick-chance)
[0.81, 0.19], per row
...]
predict_proba returns two columns per row.
Column 0 is the well-chance, written P(B) (B = benign = well).
Column 1 is the sick-chance, written P(M) (M = malignant = sick).
The two add to 1.
For the ROC curve you want the sick-chance, which is column 1.
The code that sweeps it into a curve is at the end of the post.
PLOTTING IT
When you plot it (code at the end), the grey 45-degree diagonal is the reference line.
A machine that assigns random scores sits right on that diagonal.
So any curve above the diagonal beats random guessing.
And the further it bulges toward the top-left corner, the cleaner the separation.
PRACTICAL READING
AUC ~= 0.99 the two clouds are cleanly separated; almost any cutoff works
AUC ~= 0.85 good separation; choice of cutoff matters; tune recall vs precision
AUC ~= 0.70 noisy; weak signal; more columns or a different machine might help
AUC ~= 0.50 no signal at all; the 30 columns carry no information about the bin
>> NOTE: THE TRADE CURVE AND AUC DO NOT PICK A CUTOFF FOR YOU
They tell you how well-separated the two groups are.
But the choice of cutoff is a clinical or business decision.
That choice is how much false-alarm (well lumps called sick) you accept to buy a given
catch-rate (sick lumps caught over all truly sick).
In cancer screening, a doctor reading a curve of AUC 0.97 might pick the cutoff that
gives 99% recall even if the false-alarm rate climbs to 15%.
So the curve shows the available deals; the doctor picks one.
Nothing above needed a computer -- only pencils, clerks, and patience.
This last section is for the day you meet a computer: the same work, spoken in Python.
First the 6-person sweep and AUC trapezoid from the worked example, hard-coded:
6-PERSON PILE: 3 SICK (A=0.97, B=0.88, C=0.72), 3 WELL (D=0.45, E=0.22, F=0.11)
SWEEP 7 CUTOFFS; AT EACH ONE COUNT CAUGHT, MISSED, ALARM, CLEAR
CUTOFF 0.99: NOBODY ABOVE -> TPR=0/3=0, FPR=0/3=0
tpr_1, fpr_1 = 0/3, 0/3 # 0.0, 0.0
CUTOFF 0.90: A ABOVE (SICK) -> CAUGHT=1, MISSED=2, ALARM=0, CLEAR=3
tpr_2, fpr_2 = 1/(1+2), 0/(0+3) # 0.333, 0.0
CUTOFF 0.80: A,B ABOVE -> CAUGHT=2, MISSED=1, ALARM=0, CLEAR=3
tpr_3, fpr_3 = 2/(2+1), 0/(0+3) # 0.667, 0.0
CUTOFF 0.60: A,B,C ABOVE -> CAUGHT=3, MISSED=0, ALARM=0, CLEAR=3
tpr_4, fpr_4 = 3/(3+0), 0/(0+3) # 1.0, 0.0
CUTOFF 0.40: A,B,C,D ABOVE -> CAUGHT=3, ALARM=1
tpr_5, fpr_5 = 3/(3+0), 1/(1+2) # 1.0, 0.333
CUTOFF 0.20: A,B,C,D,E ABOVE -> CAUGHT=3, ALARM=2
tpr_6, fpr_6 = 3/(3+0), 2/(2+1) # 1.0, 0.667
CUTOFF 0.00: ALL ABOVE -> CAUGHT=3, ALARM=3
tpr_7, fpr_7 = 3/(3+0), 3/(3+0) # 1.0, 1.0
print(round(tpr_2,3), round(fpr_2,3)) # 0.333 0.0
print(round(tpr_5,3), round(fpr_5,3)) # 1.0 0.333
AUC BY TRAPEZOIDAL RULE: SUM OF (FPR_GAP) X (TPR_AVG) STRIPS
DOTS IN ORDER: (FPR1,TPR1), (FPR2,TPR2) ... (FPR7,TPR7)
strip_1 = (fpr_2-fpr_1)*(tpr_1+tpr_2)/2 # (0-0)*(0+0.333)/2 = 0.0
strip_2 = (fpr_3-fpr_2)*(tpr_2+tpr_3)/2 # 0.0
strip_3 = (fpr_4-fpr_3)*(tpr_3+tpr_4)/2 # 0.0
strip_4 = (fpr_5-fpr_4)*(tpr_4+tpr_5)/2 # (0.333-0)*(1+1)/2 = 0.333
strip_5 = (fpr_6-fpr_5)*(tpr_5+tpr_6)/2 # (0.667-0.333)*(1+1)/2 = 0.333
strip_6 = (fpr_7-fpr_6)*(tpr_6+tpr_7)/2 # (1-0.667)*(1+1)/2 = 0.333
auc = strip_1+strip_2+strip_3+strip_4+strip_5+strip_6 # 1.000
print(round(auc,3)) # 1.000 -- sick scores all above well scores, perfect ranking
The AUC is 1.0 because all 3 sick chances (0.97, 0.88, 0.72) sit above all 3 well chances
(0.45, 0.22, 0.11) -- perfect separation. The toolbox block below runs the same sweep on
the full Wisconsin exam:
The ROC sweep needs a continuous chance-score, not a 0/1 label.
So grab predict_proba's column 1, which is the sick-chance (the 0-to-1 number).
Feed it to roc_curve.
Then plot the curve against the diagonal.
>> NEW TO PYTHON? Each named once:
array[:, 1] -- take column 1 from every row (slicing a 2-D array)
a, b, c = func() -- unpack several returned values into separate names
f'AUC = {x:.3f}' -- an f-string: drop a value into text, here to 3 decimals
y_proba = log_reg.predict_proba(X_test_scaled)[:, 1] # sick-chance per row
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
roc_auc = auc(fpr, tpr)
COLUMN 0 FLIPS THE CURVE BELOW THE DIAGONAL
Column 0 is the well-chance; column 1 is the sick-chance.
If you pass column 0 by mistake, sick lumps score LOW and well lumps score HIGH.
Which means the ranking is backwards.
So the ROC curve dips below the diagonal and AUC falls below 0.5.
Then the machine looks worse than a coin flip.
Always use column 1.
And the plot -- the curve, plus the grey diagonal for reference:
plt.figure(figsize=(7, 5))
plt.plot(fpr, tpr, label=f'AUC = {roc_auc:.3f}')
plt.plot([0, 1], [0, 1], linestyle='--', color='grey', label='Random guess')
plt.xlabel('FPR (false-alarm rate)')
plt.ylabel('TPR (catch-rate)')
plt.title('Trade Curve')
plt.legend()
plt.show()
Plain term used above Standard label
------------------------------------- -------------------------------------------
catch-rate (TPR) true positive rate / sensitivity / recall
false-alarm (FPR) false positive rate / (1 - specificity)
trade curve ROC curve (receiver operating char.)
area under trade curve AUC (area under the ROC curve)
chance-score posterior probability / predict_proba
pick-higher reading Wilcoxon-Mann-Whitney: P(score+ > score-)
trapezoidal rule numerical integration used by auc()
----------------------------------------------------------------------------------------------
IN THIS CHAPTER (Chapter 3 -- Sorting Into Bins):
Part 1 -- The S-Curve, the Four-Box Table .
Part 2 (this post) .
Part 3 -- Leash and Cloud .
Part 4 -- Picking Settings, Skewed Piles
<- Back to all posts
----------------------------------------------------------------------------------------------
home . source on GitHub
==============================================================================================