Winning Kaggle the Reproducible Way

Chapter 38

Calibration and Post-Processing

Postprocessing is another fitted model: give it a development set and a separate assessment.

A calibrator is fitted on development predictions and judged elsewhere. One demonstration measures how a two-parameter Platt calibrator and a flexible isotonic calibrator behave as the development set grows, on the rows they were fitted to and on new rows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Platt or isotonic calibration, by the size of the development set

How large must the development set be before isotonic calibration beats the simpler Platt calibrator on new rows?

Platt scaling fits two numbers, so it cannot chase noise, but it cannot follow a bent calibration curve either. Isotonic regression fits a free monotone step function: with few rows it memorizes them and looks excellent on its own development set; with many rows it follows the true curve and wins. Both are monotone, so neither improves ranking.

Equation: E C E equals the sum over bins b of n b over n times the absolute value of y bar b minus p bar b

Scroll sideways for the whole equation

n_b is the number of rows in probability bin b, n the total, y-bar_b the observed event rate and p-bar_b the mean predicted probability in that bin.

Predict first. With 50 development rows, which calibrator has the lowest log loss on its own development rows, and which on new rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Platt or isotonic calibration, by the size of the development set. Left: reliability curves on assessment rows for no calibration, Platt and isotonic with 200 development rows. Right: log loss on development rows and on new rows; isotonic scores 0.509 on its own rows and 0.589 on new rows, Platt 0.566.
Development rows used to fit the calibrator: 200
Constructed example: seeded synthetic data, a naive Bayes base model and two calibrators, measured by the chapter activity.

Calculated values

No calibration, new rows
0.959
Platt, new rows
0.566
Isotonic, new rows
0.589
Isotonic, own rows
0.509
Isotonic beats Platt
20% of draws

With 200 development rows, calibration cuts log loss on new rows from 0.959 to 0.566 with Platt (0.959 - 0.566 = 0.393) and 0.589 with isotonic, so Platt is better on new rows (isotonic won 20% of 30 draws). On its own development rows isotonic scores 0.509, 0.080 better than it does on new rows. AUC moves from 0.768 to 0.768 (Platt) and 0.760 (isotonic).

Worked steps

  1. Gain from Platt on new rows: 0.959 - 0.566 = 0.393.
  2. Isotonic minus Platt on new rows: 0.589 - 0.566 = +0.023.
  3. Isotonic optimism: 0.589 - 0.509 = 0.080 (new rows minus own rows).
  4. Platt optimism: 0.566 - 0.552 = +0.015.

Use the idea

Fit every calibrator on development predictions, score it on a separate assessment partition, and compare it with a simpler calibrator and with no calibration before keeping it.

Where the conclusion applies

Constructed data, one overconfident base model and probabilities kept inside [0.001, 0.999]. The crossover size depends on how bent the true calibration curve is; with a curve Platt can represent exactly, isotonic needs even more rows to catch up.

Check your understanding: Why does the AUC barely change after calibration, even when log loss improves by a third?
Both calibrators are monotone, so they keep the base model's ordering of rows. AUC measures only ordering. Platt leaves it exactly unchanged; isotonic can merge nearby probabilities into ties, which slightly lowers AUC. The log-loss gain comes from fixing overconfidence, not from ranking better.

Chapter 38 source: section "Fit Calibration on the Correct Predictions".