Winning Kaggle the Reproducible Way

Chapter 51

Time Series Cross-Validation

At a forecast origin, train only on labels that would already exist, and watch which recipes the shortcut flatters.

A row observed before the origin is not usable if its label arrives after it. One demonstration varies the label delay and compares a backtest that respects label availability with one that ignores it, for a recent-history recipe and an expanding-history recipe.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Backtests that respect and ignore label availability, for two recipes, by label delay

How much does a backtest that ignores label availability flatter each recipe, and does it still rank the recipes correctly?

The naive backtest never looks at the delay, so its error is the same at every value. The honest backtest cannot use the newest rows, and the world has moved by the time the model is used, so its error rises with the delay. A recipe that relies on the most recent rows loses the most, which is why the shortcut flatters recency-heavy recipes most. Here it ranks the 30-day window first at every delay, which is right only when labels arrive at once; respecting availability puts the expanding recipe ahead from a 10-day delay on.

train on row i iff d_i + L < v

Scroll sideways for the whole equation

d is the day a row is observed, L the label delay and v the forecast origin. A row is usable only if d + L < v. MAE is the mean absolute error on the next 30 days; optimism is the honest backtest's MAE minus the naive backtest's.

Predict first. With a 30-day label delay, which recipe does the backtest that ignores label availability prefer?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Backtests that respect and ignore label availability, for two recipes, by label delay. Two panels at a 30-day label delay. Left, mean absolute error of the 30-day-window and expanding recipes under a backtest that ignores label delay and one that respects it, with a dashed noise floor at 0.798. Right, the optimism of the naive backtest for each recipe, 0.322 for the short window and 0.123 for the expanding recipe.
Label delay (days between observing a row and receiving its label): 30
Constructed example: seeded synthetic daily regression data with drifting coefficients and ridge models, measured by the chapter activity.

Calculated values

Naive MAE, 30-day window
1.919
Honest MAE, 30-day window
2.241
Naive MAE, expanding
1.957
Honest MAE, expanding
2.080
Optimism, 30-day window
+0.322
Optimism, expanding
+0.123

With a 30-day delay the naive backtest scores the 30-day window 1.919 and the expanding recipe 1.957, a gap of -0.038 (standard error 0.013), so it prefers the 30-day window; the honest backtest reverses that, with the expanding recipe ahead by 0.161 (standard error 0.015). Respecting availability raises the short window's error to 2.241: 2.241 - 1.919 = 0.322 of optimism, against 0.123 for the expanding recipe. With fixed coefficients the two rules differ by 0.001 MAE, so the effect comes from a relationship that changes.

Worked steps

  1. Short window optimism: 2.241 - 1.919 = +0.322 (standard error 0.015).
  2. Expanding optimism: 2.080 - 1.957 = +0.123 (standard error 0.005).
  3. Naive ranking gap, short minus expanding: 1.919 - 1.957 = -0.038.
  4. Honest ranking gap, short minus expanding: 2.241 - 2.080 = +0.161.

Use the idea

Record for every row when its label becomes available, build the training set at each origin from labels that exist by then, and compare recipes only under that rule.

Where the conclusion applies

Constructed linear data with drifting coefficients, ridge regression instead of a gradient-boosted model and five origins sharing the same series; fold results are correlated, so the standard errors describe the 160 series, not a confidence interval for a competition. In the control world with fixed coefficients the two rules differ by at most 0.002 MAE, so the effect needs a relationship that changes. The sizes depend on the drift speed chosen here.

Check your understanding: The naive error is 1.919 for the short window at every delay. What does that constancy tell you, and which recipe does the shortcut flatter more?
The naive backtest cannot see the delay at all, so its number carries no information about it. The honest MAE of the short window grows from 1.919 at no delay to 2.241 at 30 days, an optimism of 0.322, against 0.123 for the expanding recipe, because the short recipe depends only on the newest rows, which are exactly the ones whose labels have not arrived.

Chapter 51 source: section "Define the Forecast Origin".