Winning Kaggle the Reproducible Way

Chapter 4

Tracking the CV-LB Gap

A gap is a discrepancy to investigate; judge a change with paired differences, not with two raw scores.

A public score is a sample. One demonstration measures how far an unchanged model's gap wanders from sampling alone, how often a truly better candidate looks worse on the public rows, and how paired and unpaired reading of the same rows differ.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Sampling-only gaps and a small true gain, by public sample size

With no shift and no leak, how far does the CV-LB gap move from sampling alone, and can a public sample of a given size reveal a candidate that is truly better by a small amount?

An unchanged model has no gap pattern to find, yet its gap moves by sampling alone, by a lot at small m. The candidate's real gain is much smaller than that movement, so a raw reading (did the public score go up or down?) fails often. The two models make nearly the same errors on each row, so their per-row difference has a far smaller spread than either score. Pairing uses that; reading the two scores as independent does not.

Equation: gap equals public loss minus local loss

Equation: d i is the baseline row loss minus the candidate row loss; the standard error is the standard deviation of d i over the square root of m

Scroll sideways for the whole equation

gap = public loss - local loss (the chapter's definition for a loss). d_i = l_i(baseline) - l_i(candidate) is the per-row paired loss difference, m the public rows, and SE = sd(d_i) / sqrt(m) its standard error; a positive d means the candidate is better.

Predict first. With 1,000 public rows, how often does a candidate that is truly better by about 0.002 log loss look worse than the baseline on the public sample?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Sampling-only gaps and a small true gain, by public sample size. Left: histogram of the candidate's public improvement over 500 public samples of 1,000 rows, with a dashed line at the true gain 0.0021. Right: bars for how often the candidate looks worse (16%) and how often the unpaired and paired readings find the gain.
Rows in the public sample: 1,000 rows
Constructed example: seeded synthetic rows and logistic models, measured by the chapter activity. The numbers describe this generator only.

Calculated values

Gap spread (public sampling)
0.014
Gap spread (development side)
0.009
True gain
0.0021
Better candidate looks worse
16%
Paired detects
19%
Unpaired detects
0%

With 1,000 public rows an unchanged model's gap wanders by a standard deviation of 0.014 from public sampling alone, and by 0.009 from the development side, against a true gain of 0.5496 - 0.5475 = 0.0021. The truly better candidate looks worse on 16% of public samples. Read as two independent scores the gain is found in 0% of samples (standard error 0.020); read as a paired difference it is found in 19% (standard error 0.002).

Worked steps

  1. True gain: 0.5496 - 0.5475 = 0.0021 (baseline minus candidate loss on 60,000 hidden rows).
  2. Sampling-only gap spread, public side: 0.014; development side: 0.009.
  3. Share of public samples where the better candidate looks worse: 16%.
  4. Share detected at 1.96 standard errors: paired 19%, unpaired 0%.

Use the idea

Do not read a gap change of the size shown here as a diagnosis. Compare candidate and baseline on the same rows with a paired difference and its standard error, and ask for more rows or blocks before deciding.

Where the conclusion applies

Constructed data with no shift, no leak and no selection, one logistic model pair, 20 competitions of 300 public draws each. Real gaps add population shift and selection on top of the sampling spread measured here. The unpaired rule is a deliberately naive reading of two public numbers, shown to contrast with the paired one. The local-side spread comes from a 3,000 row development set and is measured across competitions.

Check your understanding: Why can the paired comparison detect a real gain that the two raw public scores cannot, using the same public rows?
Both models make nearly the same error on each row, so the per-row difference between them varies far less than either score does: the paired standard error at 1,000 rows is 0.002 against 0.020 when the two scores are treated as independent. A paired reading found the gain in 19% of public samples at 1,000 rows and 94% at 16,000; the unpaired reading found it in almost none. Even paired, 1,000 rows is too few to settle a 0.002 gain.

Chapter 4 source: section "The Three Gap Patterns".