Winning Kaggle the Reproducible Way

Chapter 13

Hyperparameter Tuning with Optuna

A tuned configuration's winning score is a development result until a frozen recipe is assessed elsewhere.

A hyperparameter search reports the best score it found. One demonstration runs a random search on boosted trees and measures that score against what the frozen winner earns on rows reserved before the search.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A random search's best score, against the frozen winner on reserved rows

As the trial budget grows, how much of the search's reported improvement shows up on rows the search never used?

Each trial is another noisy draw of a configuration's development score, so the best-so-far score can only rise. Part of every rise is the winner's luck on these folds, which does not carry to new rows. The gain on reserved rows grows more slowly and flattens, so the optimism tends to widen with the budget. With a single trial nothing is selected, and the two gains differ only by noise.

Equation: optimism at T trials equals the search gain at T minus the assessment gain at T

Scroll sideways for the whole equation

G_search(T) is the best development score found in T trials minus the default configuration's development score, G_assess(T) is the frozen winner's AUC on the reserved rows minus the default's, and the optimism is their difference.

Predict first. After 16 random-search trials, how does the gain the search reports over the default compare with the gain on reserved rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A random search's best score, against the frozen winner on reserved rows. Left: mean AUC gain over the default against trials run, for the search's best score (0.032 at the end) and the frozen winner on reserved rows (0.008), with thin lines per dataset. Right: the two final gains with a dot per dataset.
Search trials: 16
Constructed example: seeded synthetic data, a random search over boosted trees, measured by the chapter activity.

Calculated values

Trials
16
Search-reported gain
+0.032
Gain on reserved rows
+0.008
Optimism
0.024
Datasets helped
75%

After 16 trials the search reports a gain of +0.032 AUC over the default; the frozen winner earns +0.008 on 4,000 reserved rows, so 0.032 - 0.008 = 0.024 of the reported gain is optimism. The winner beat the default on reserved rows in 75% of the 12 datasets.

Worked steps

  1. Optimism: 0.032 - 0.008 = 0.024 AUC (search gain minus reserved-row gain).
  2. Share of the reported gain that carried over: 0.008 / 0.032 = 0.248.
  3. Datasets where the tuned winner beat the default on reserved rows: 75%.

Use the idea

Report a tuning result as the frozen recipe's gain on reserved rows over the baseline, with the budget and cost. Stop searching when the reserved-row gain stops moving, not when the development score stops climbing.

Where the conclusion applies

Constructed data, random search in place of Optuna's TPE sampler, scikit-learn HistGradientBoostingClassifier in place of LightGBM, and 3 folds rather than 5; Optuna and LightGBM are not installed here. The chapter notes that random and TPE searches both remain valid comparators. A 300-row search set is deliberately small, which makes the development score noisy; with more rows the optimism shrinks.

Check your understanding: The development score can only rise with more trials. Why is that not evidence that the tuned model keeps getting better?
The score is a running maximum of noisy estimates, so it is non-decreasing by construction. Going from 4 to 16 trials raises it by 0.014 on the search folds but changes the reserved-row gain by less than 0.001; the extra development gain is mostly selection luck, which is why the chapter asks for a frozen recipe and an untouched comparison.

Chapter 13 source: section "The Optuna Pattern".