Winning Kaggle the Reproducible Way

Chapter 61

Playground Series Strategy

Small leaderboard differences are noise plus selection: keep a supported candidate, not the public leader.

The public leaderboard is a small sample, and picking its leader among many near-equal candidates selects luck. One demonstration measures how much of the leader's visible advantage survives on private rows, and how a local cross-validation pick compares, as the number of screened candidates grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

The public leader's advantage, shown and earned, by number of candidates screened

How much of the public leader's visible advantage is still there on private rows as more near-equal candidates are screened?

Each candidate's public RMSE is its true quality plus sampling noise from only 400 rows. Taking the minimum over more candidates picks more noise: the advantage the leaderboard shows keeps growing with the candidate count, while the advantage that exists on private rows or fresh rows changes far less (up or down by about 0.01 across seeds), so the noise in the visible lead keeps growing. Cross-validation uses 1,500 rows, so it is usually a quieter judge here, but it is also one more selection, so its pick loses ground too.

Equation: The public lead is the median over candidates of the public RMSE minus the public RMSE of the pick

Scroll sideways for the whole equation

s_c is candidate c's RMSE (lower is better), the superscripts pub and priv mark public and private rows, median over c is taken across the screened candidates and pick is the candidate with the lowest public RMSE.

Predict first. With 100 candidates screened, the public leader looks 0.062 RMSE better than the median candidate on the public rows. How much of that advantage shows up on private rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: The public leader's advantage, shown and earned, by number of candidates screened. Left: bars of the public leader's RMSE lead over the median candidate with 30 candidates: 0.052 shown on public rows, 0.018 earned on private rows and 0.017 on fresh rows. Right: regret on fresh rows for picking by public score (0.017), by cross-validation (0.009) and at random (0.042).
Candidates screened against the public leaderboard: 30
Constructed example: seeded synthetic regression data and 3 to 100 ridge candidates per competition, measured by the chapter activity.

Calculated values

Lead shown on public rows
0.052
Lead earned on private rows
0.018
Lead earned on fresh rows
0.017
Public leader also private leader
23% of splits
Regret: public / CV / random
0.017 / 0.009 / 0.042

With 30 candidates the public leader looks 0.052 RMSE better than the median candidate on the public rows, but earns 0.018 on private rows and 0.017 on fresh rows: 0.052 - 0.018 = 0.034 of the visible lead was selection noise, and 35% of it is real. The public leader is also the private leader in 23% of splits. Against the best candidate, picking by public score costs 0.017 RMSE on fresh rows, picking by cross-validation 0.009 and picking at random 0.042.

Worked steps

  1. Noise in the visible lead: 0.052 - 0.018 = 0.034 (shown minus earned on private rows).
  2. Share of the visible lead that is real: 0.018 / 0.052 = 0.351.
  3. Cross-validation against the public leader: 0.017 - 0.009 = 0.008 less regret.
  4. Public leader against a random pick: 0.042 - 0.017 = 0.025 less regret.

Use the idea

Treat a small public lead as weak evidence, judge each change on paired local folds, and fill the final submission slots with candidates that have supported local evidence rather than only the public leader.

Where the conclusion applies

Constructed data with ridge variants standing in for the many near-equal models of a Playground competition, and one fixed public share (20%). Cross-validation had less regret than the public pick in every state here, by 0.002 to 0.01, but under another seed the two were tied (cross-validation very slightly worse) in one state; it is a property of the training set (1,500 rows) being much larger than the public set (400 rows). With a relatively larger public set or a leaking cross-validation the order could change. Costs here are RMSE units on a target with noise standard deviation 3.

Check your understanding: If the public lead overstates so much, why not ignore the public score and pick at random?
The public score still carries signal: with 100 candidates, picking the public leader is 0.019 RMSE worse than the best candidate on fresh rows, against 0.048 for a random pick. The leader is the best candidate only 13% of the time, though, and the leader on the public rows is also the leader on private rows only 8% of the time. Use the public score as one weak vote, not as the decision.

Chapter 61 source: section "Reading the Leaderboard: Signal vs. Noise".