Winning Kaggle the Reproducible Way

Chapter 19

March Mania: Prediction at Scale

Protect the forecast boundary, select on development seasons, and prove the cutoff with a label-flip test.

The chapter's constructed season forecast is rerun forty times while its history is allowed to peek at later seasons. One demonstration measures how little the score comparison reveals and how clearly the chapter's label-flip test does.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Leaked later seasons: what the scores hide and the label-flip test shows

If the team history quietly includes later seasons, would the development score tell you, and would the label-flip test?

With L = 0 the history is exactly what exists at each season's start, so reversing the season 7 and 8 labels cannot move any development score: the test passes. With L at least 1, the history for the development seasons reads later outcomes, so the development score moves when those labels change. The leak improves the development score by only a few thousandths, and later seasons are harder in this generator whatever the pipeline does, so the gap between the development and forecast scores is no reliable signal: with one to three leaked seasons it changes by less than 0.004 from the legal pipeline's gap, too little to tell apart from season difficulty and selection noise.

Equation: Brier score equals the mean over n games of the squared difference between predicted probability p i and outcome y i

Equation: the history H s for season s holds outcomes of seasons t earlier than s, plus, in a leaky build, the L seasons after s

Scroll sideways for the whole equation

p_i is the predicted probability and y_i the outcome of game i, n the games scored, s a game's season, L the number of later seasons visible in its team history and H_s the set of outcomes used to build its features.

Predict first. With one later season visible in the history, how far does reversing the season 7 and 8 labels move the development Brier of the selected recipe, on average?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Leaked later seasons: what the scores hide and the label-flip test shows. Left, bars of the development estimate 0.2257 and the forecast Brier 0.2359 with dashed lines for the best possible Brier; right, the change in development Brier after flipping later labels for forty fixtures, mean 0.0038.
Later seasons included in each game's team history: 1: one later season
Constructed example: forty seeded synthetic fixtures from the chapter's generator, logistic regression and a boosted tree, measured by the chapter activity.

Calculated values

Development estimate
0.2257
Forecast of seasons 7-8
0.2359
Label-flip change
0.0038
Recipe changed by flip
0.075
Chapter fixture, logistic dev Brier
0.210300

With 1 later season(s) in the history, the development estimate averages 0.2257 and the forecast of seasons 7 and 8 scores 0.2359, so 0.2359 - 0.2257 = 0.0102. Against the legal pipeline the estimate moves -0.0026 (standard error 0.0004). The label-flip test fails: reversing the season 7 and 8 labels moves the development Brier by 0.0038 on average and changes the selected recipe in 0.075 of fixtures. The chapter's own fixture moves by 0.0034.

Worked steps

  1. Development estimate: 0.2257; forecast of seasons 7 and 8: 0.2359.
  2. Gap: 0.2359 - 0.2257 = 0.0102 (later seasons are harder here: best possible 0.216 then 0.220).
  3. Estimate against the legal pipeline: -0.0026.
  4. Label-flip change in the development Brier: 0.0038 - 0 = 0.0038 (a legal pipeline gives 0).

Use the idea

Run the label-flip test on the real pipeline: change the assessment labels, rebuild everything and confirm that the development scores, the selected recipe and the frozen model do not change.

Where the conclusion applies

Constructed fixtures in which team strengths drift and the spread of strengths shrinks, so later seasons are inherently harder; two fixed candidates and a single feature. In a real task the size of a leak depends on how much the later outcomes say about the earlier rows. The flip test measures a dependency, not its cost.

Check your understanding: The leaky pipelines report a development Brier barely lower than the legal one. Why is the label-flip test still reliable?
The score comparison asks whether a number looks too good, and the leak moves that number by only 0.0026 at L = 1, smaller than the 0.0095 gap that harder later seasons and ordinary selection produce with no leak at all. The flip test asks a yes-or-no question about dependence: a legal pipeline moves by exactly 0.000, so any movement proves that later labels reach the development scores.

Chapter 19 source: section "Run the Complete Constructed Forecast".