Winning Kaggle the Reproducible Way

Chapter 18

Final Submission Selection

Choose a recoverable champion from credible evidence, and know how much a small public split can tell you.

Among close candidates, a small public leaderboard is noisy evidence. One demonstration measures how often public-only, local-only and pooled evidence each pick the truly best candidate as the public share changes.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Choosing among twelve close candidates: public score, local score or both

How much public evidence does it take before the public leaderboard picks as well as a local validation set?

Every rule scores candidates on the same rows, so each comparison is paired, but a small set of rows still leaves differences of a few thousandths of log loss within noise. The public rule is the most exposed, because the candidate that wins a noisy contest is partly the luckiest one, and its public score flatters it. More rows, from any source, shrink both effects. Pooling the public and local rows uses all the evidence and picks at least as well as either alone.

Equation: s c is the mean of the per-row loss of candidate c over the evidence rows S; the submitted candidate c star minimizes s c; regret is the true loss of c star minus the smallest true loss

Scroll sideways for the whole equation

S is the set of rows an evidence source uses, loss_c,i the log loss of candidate c on row i, s_c the mean of loss_c,i over S, c* the submitted candidate and L_c the true loss of candidate c on the large pool.

Predict first. With 10% of the hidden rows public (200 rows), how often does the best public score belong to the truly best of the twelve candidates?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Choosing among twelve close candidates: public score, local score or both. Two panels with three bars each for public-only, local-only and pooled selection at a 200-row public set; left shows regret, right shows the optimism of the selected candidate's score. The public rule has regret 0.00162.
Share of the 2,000 hidden test rows that are public: 10%: 200 public rows
Constructed example: five seeded synthetic tasks with twelve fitted gradient-boosting variants, measured by the chapter activity.

Calculated values

Public rule picks the best
0.594
Local rule picks the best
0.796
Pooled rule picks the best
0.818
Public rule regret
0.00162
Public winner's optimism
0.00277

With 200 public rows out of 2000, the public rule submits the best of 12 candidates in 0.594 of splits and the 1000-row local rule in 0.796. Mean regret is 0.00162 against 0.00044, so 0.00162 - 0.00044 = 0.00118. Local rows pick better than the 200 public rows. Pooling gives 0.00039. The public winner's score is 0.00277 better than its true loss, so a winning public score overstates it.

Worked steps

  1. Public rows only: best candidate picked in 0.594 of splits, regret 0.00162.
  2. Local rows only: best candidate picked in 0.796 of splits, regret 0.00044.
  3. Pooled rows: best candidate picked in 0.818 of splits, regret 0.00039.
  4. Public minus pooled regret: 0.00162 - 0.00039 = 0.00123.

Use the idea

Keep a local validation set that no candidate was fitted or tuned on, compare candidates on identical rows, and treat the public leaderboard as one more block of rows whose weight grows with its size.

Where the conclusion applies

Constructed data and one model family. The local rows are assumed to be fresh rows from the same population as the hidden test, which real local validation only approximates; a local score that guided tuning would be more optimistic than here. The public rows are read once, at selection; a public score that has already guided many submissions is no longer fresh evidence, which is why the chapter asks you to record that influence. This is a local stress test, not a reproduction of any hidden split. The crossover near 50% is where the public set has as many rows as the local set (1,000), so it moves with the local set's size.

Check your understanding: Why does the pooled rule beat both single sources, and why does the public rule only match the local one near a 50% public share?
Pooling uses every row once, so it has the most evidence at every share. The public and local rules each use only their own rows, and they tie when the public set (1,000 rows at 50%) is as large as the local set. Even the best rule does not pick the true best candidate every time, so the private ranking of the pick still moves.

Chapter 18 source: section "Simulating Public/Private Split Stability".