Winning Kaggle the Reproducible Way

Chapter 25

The Continuous Learning System

A logged gain is evidence only when it is larger than the noise that produced it.

A learning record keeps a gain such as +0.002 AUC. One demonstration measures the noise behind such a note: the fold-seed spread, the larger spread that seeds cannot remove, and what a seed-based threshold lets through.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A logged CV gain against the seed noise floor

How much of the noise behind a logged gain does averaging more fold seeds remove, and what does a seed-based threshold then keep?

Averaging k seeds divides the fold-seed spread by sqrt(k), so a threshold built from seed noise keeps shrinking. But the logged gains of pure-noise features also vary because the sample is finite, and re-seeding the folds does not touch that part. The floor falls faster than the real noise, so more seeds let more noise features through. They also let more real features through, but the share of kept notes that are real falls, and at every setting the gains of kept notes overstate their true gains.

Equation: the standard error for a note averaging k seeds is the seed standard deviation divided by the square root of k; keep a note if its average gain exceeds twice that standard error

Scroll sideways for the whole equation

g_s is the paired AUC gain of one candidate under fold seed s, sd_seed the standard deviation of g_s across seeds, SE = sd_seed / sqrt(k) the standard error of a note that averages k seeds and the seed floor is 2 SE.

Predict first. Moving from a single run to 10 averaged fold seeds shrinks the seed standard error about threefold. What happens to the share of pure-noise features that a gain-above-twice-the-seed-error rule keeps?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A logged CV gain against the seed noise floor. Left, bars of the seed standard error (0.00021) and the spread of pure-noise features' gains (0.00075) for notes averaging 10 seed(s), with lines for the true gain and the seed floor; right, the share of real and noise features kept by two rules.
Fold seeds averaged in each logged note: 10 seeds
Constructed example: thirty-two seeded synthetic competitions with ridge-regularized linear models, measured by the chapter activity.

Calculated values

Seed standard error of a note
0.00021
Spread of noise features' gains
0.00075
Seed floor (2 SE)
0.00042
Noise features kept by the seed floor
0.091
Real features kept by the seed floor
0.688

A note averaging 10 seed(s) has a seed standard error of 0.00021, so the seed floor is 2 x 0.00021 = 0.00042. Across 352 pure-noise features the logged gains still spread by 0.00075, 3.6 times the seed error. The seed-floor rule keeps 0.091 of noise features and 0.688 of real ones; any positive gain keeps 0.170 of noise features. Kept notes log 0.00240 on average against a true 0.00094: 0.00240 - 0.00094 = 0.00146 of overstatement.

Worked steps

  1. Seed standard deviation of one run: 0.00066; with 10 seed(s) the standard error is 0.00021.
  2. Spread of the noise features' logged gains: 0.00075 (seed error 0.00021).
  3. Seed-floor rule: keeps 0.688 of real and 0.091 of noise features; precision 0.77.
  4. Kept notes: logged 0.00240 - true 0.00094 = 0.00146.

Use the idea

Treat a logged gain as a measurement with a noise level you estimate from pure-noise candidates and fresh rows, not from re-seeding alone; store the seeds, fold IDs and the true holdout result with the note.

Where the conclusion applies

Constructed data with a linear model and a deliberately small true effect (about +0.0016 AUC). Candidates are independent of each other. With a larger true effect every rule keeps more real features and the noise features matter less; with correlated candidates the noise is worse. Averaging seeds still helps: it removes the seed part of the noise and makes a note reproducible.

Check your understanding: A note logs +0.002 averaged over 10 seeds, about ten times the seed standard error. Why is it still weak evidence?
Averaging seeds removes only the seed noise. Pure-noise features still log gains that spread by 0.0008 across candidates, and the 10-seed floor is only 0.0004. The kept notes' logged gains average 0.0024 while their true gains average 0.0009, so a note should be confirmed on rows the seeds never touched, and recorded with its noise estimate.

Chapter 25 source: section "Preserve Failures and Reproduction Conditions".