Winning Kaggle the Reproducible Way

Chapter 59

Threshold Optimization

A tuned threshold's search score is a development result: freeze it and measure it on different labels.

Threshold search fits the labels it is scored on. One demonstration tunes an F1 threshold on development sets of different sizes, freezes it, and measures how much the development score overstates the score on 50,000 fresh rows, and what happens when the prevalence changes.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Tuned F1 threshold: development score against fresh-label score, by development-set size

How far does the development F1 of a tuned threshold overstate its F1 on fresh labels, and how does that depend on the size of the development set?

A search over every cut of a small development set picks the threshold that happens to rank its handful of positives best, so its score is inflated and the threshold itself is noisy. More rows shrink both. Tuning still beats the default 0.5 because the default is badly matched to a rare class. A frozen threshold also carries the prevalence it was tuned at: with 1% positives instead of 5% it no longer beats the default 0.5 (here it scores below it), and with 20% it falls well short of the best cut for that audit.

Equation: F1 at threshold t equals twice the true positives over twice the true positives plus false positives plus false negatives; optimism equals the maximum over t of the development F1 minus the audit F1 at the chosen threshold t hat

Scroll sideways for the whole equation

F1 = 2TP / (2TP + FP + FN) for the rows called positive at threshold t. The development score is the maximum of F1 over t on the development rows; the audit score is F1 at that frozen t on fresh rows. Optimism is the difference.

Predict first. With a development set of 100 rows, by how much does the tuned threshold's development F1 exceed its F1 on 50,000 fresh rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Tuned F1 threshold: development score against fresh-label score, by development-set size. Left: with 100 development rows, the tuned threshold scores 0.502 on development rows and 0.315 on fresh rows; the default 0.5 scores 0.215 and the best possible threshold 0.361. Right: F1 on audits with 1%, 5% and 20% positives for the default, the frozen threshold and the best threshold for each audit.
Rows in the development set used to tune the threshold: 100 (about 5 positives)
Constructed example: seeded synthetic data with 5% positives, a logistic regression and 300 development sets per size, measured by the chapter activity.

Calculated values

Development F1 (tuned)
0.502
Fresh-rows F1 (same threshold)
0.315
Optimism
0.187
Default 0.5 F1
0.215
Tuned beats default
93% of draws

With 100 development rows, the tuned threshold reaches an F1 of 0.502 on those rows but 0.315 on 50,000 fresh rows, so 0.502 - 0.315 = 0.187 is the optimism. The default 0.5 scores 0.215 on the fresh rows, so tuning gains 0.315 - 0.215 = 0.100 and beats the default in 93% of 300 draws. The best possible threshold scores 0.361. The tuned threshold itself varies by 0.132 (one standard deviation) around 0.225. Frozen at 5% positives and scored where only 1% are positive it gives 0.139 (default 0.5: 0.192), and at 20% positive 0.481 (best cut for that audit: 0.595).

Worked steps

  1. Optimism: 0.502 - 0.315 = +0.187 (development minus fresh rows).
  2. Gain over the default 0.5: 0.315 - 0.215 = +0.100.
  3. Shortfall against the best possible threshold: 0.361 - 0.315 = +0.046.
  4. Fold-averaged threshold minus single tuned threshold on fresh rows: 0.324 - 0.315 = +0.009.

Use the idea

Tune the threshold on development predictions, freeze it, report the frozen score on separate labels, and check it under the prevalence you expect rather than quoting the search score.

Where the conclusion applies

Constructed data with one logistic model and a stable relationship, so the audit set comes from the same generator. A development draw with no positive row is given one so a threshold can be tuned; that affects about 0.6% of draws at 100 rows. The best possible threshold is tuned on the audit labels themselves and is a ceiling, not an achievable score. The prevalence-shift audits assume the same feature distribution and only a different share of positives.

Check your understanding: The development score is badly optimistic at 100 rows, yet the tuned threshold still beats the default 0.5. Are those two facts in conflict?
No. The development score is a biased estimate of the threshold's quality, not a measure of whether tuning helps. The tuned threshold scores 0.315 on fresh rows against 0.215 for the default 0.5, and beats it in 93% of the 300 draws. The bias is the reason to freeze the threshold and score it on other labels before quoting any number.

Chapter 59 source: section "When Threshold Optimization Overfits".