Winning Kaggle the Reproducible Way

Chapter 3

Building a CV That Matches the Test Set

A split defines what unseen means; match it to how the test set is built.

Leave-one-season-out trains on later seasons, a real forecast cannot. One demonstration measures how far leave-one-season-out and past-only validation each sit from the score on the next season as the relationship drifts.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Leave-one-season-out against past-only validation, as the relationship drifts

When the relationship between features and outcomes changes from season to season, how far does each validation scheme's Brier score sit from the score on the next season?

With no drift, seasons are exchangeable and both schemes describe the next season equally well. With drift, a model trained on seasons on both sides of the held-out one has seen a close relative of its test season, so leave-one-season-out looks better than a forecast can be. Past-only validation scores a genuine forecast, so it sits closer to the next-season score, though in this generator it is still a little optimistic once there is drift.

Equation: Brier equals one over n times the sum over games i of p i minus y i, squared

Equation: past-only trains on seasons k earlier than s; leave-one-season-out trains on every season k other than s

Scroll sideways for the whole equation

Brier = the mean of (p - y)^2 over scored games, with p the predicted win probability and y the result (lower is better). Optimism is the Brier score on the next season minus the validation estimate: positive means validation looked better than the future turned out.

Predict first. At moderate drift (0.5), how does leave-one-season-out's Brier score compare with the score on the next season?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Leave-one-season-out against past-only validation, as the relationship drifts. Left: Brier score of the two validation schemes with a dashed line at the next-season score 0.196. Right: optimism of each scheme, 0.012 for leave-one-season-out and 0.006 for past-only, at drift 0.5.
Season-to-season drift (standard deviation of the weight step): 0.5: moderate
Constructed example: seeded synthetic seasons of games and a logistic model, measured by the chapter activity. The seasons are not NCAA results.

Calculated values

Leave-one-season-out Brier
0.184
Past-only Brier
0.190
Next-season Brier
0.196
Optimism, leave-one-season-out
0.012
Optimism, past-only
0.006

At drift 0.5 the final recipe scores a Brier of 0.196 on the next season. Leave-one-season-out reports 0.184, so 0.196 - 0.184 = 0.012 of optimism. Past-only reports 0.190, 0.196 - 0.190 = 0.006. Leave-one-season-out is the more optimistic scheme. Leave-one-season-out was the more optimistic in 85% of 100 histories.

Worked steps

  1. Leave-one-season-out: 0.196 - 0.184 = 0.012 (next season minus estimate).
  2. Past-only: 0.196 - 0.190 = 0.006.
  3. Difference between the schemes: 0.190 - 0.184 = 0.006 of Brier.
  4. Share of histories where leave-one-season-out was more optimistic: 85%.

Use the idea

If the test set is a future season, validate with past-only seasons and keep leave-one-season-out, if you use it, as a labelled historical grouped check.

Where the conclusion applies

Constructed league with a random-walk relationship, one logistic model and 100 histories. Real drift may be abrupt, seasonal or absent, and past-only validation trains on only four to six seasons where leave-one-season-out trains on six. At no drift the two schemes are tied and both match the next-season score.

Check your understanding: With no drift both schemes agree, so why does the chapter still tell a forecaster to use past-only seasons?
Because the agreement is a property of a stable relationship, which you cannot know in advance. Here drift of 0.5 left leave-one-season-out 0.012 from the next-season score while past-only was 0.006 away, leave-one-season-out reached 0.017 at drift 1.0, and at zero drift the two tied. Past-only costs nothing when seasons are exchangeable and protects the decision when they are not, though it is not a guarantee: it was still optimistic here.

Chapter 3 source: section "March Mania as a Special Case".