Winning Kaggle the Reproducible Way

Chapter 52

Time Series Feature Engineering

A lag is useful only when it describes information available at the forecast origin.

A feature builder that shifts by one row is correct for a next-day forecast and quietly wrong for a three-day one. One demonstration measures how far a backtest built with shift-by-one features drifts from the error of features built at the forecast origin, as the horizon grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Backtest error with shift-by-one features, against features built at the forecast origin

How much does a backtest flatter a model whose lag features read days that would not yet exist at the forecast origin?

At h = 1 every lag the shift-based builder creates is already known at the origin, so the backtest, the production score and the direct model agree. As h grows, lag 1 and its neighbours read days after the origin. The backtest keeps scoring with the true values, so its error does not move, while the direct model loses the most informative lags and its error grows, levelling off at 5 and 7 days, where it keeps the same lags (7, 14 and 21). The direct model also beats the shift-based model run as production would run it, because it was trained on columns it will actually have.

o = t - h, lag k is usable iff t - k ≤ o iff k ≥ h

Scroll sideways for the whole equation

t is the target day, h the horizon in days, o = t - h the forecast origin, and k the lag in days. A lag k is usable at the origin only when day t - k is at or before o, which holds exactly when k is at least h. MAE is the mean of the absolute forecast errors.

Predict first. At a three-day horizon, how does the backtest MAE of the shift-based pipeline compare with the MAE of the model built at the origin?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Backtest error with shift-by-one features, against features built at the forecast origin. Bars of forecast error at a 3-day horizon: shift-based backtest 4.96, shift-based run at the origin 5.95 and direct model 5.69, with a dashed line at the seasonal-naive error of 6.83.
Forecast horizon (days between the origin and the target day): 3: three days ahead
Constructed example: thirty seeded synthetic daily series per panel, ten panels and a gradient boosting model, measured by the chapter activity.

Calculated values

Shift-based backtest MAE
4.96
Shift-based, run at the origin
5.95
Direct model MAE
5.69
Hidden error (direct - backtest)
0.73
Columns that move after the origin
3 of 8

At a horizon of 3 days the shift-based backtest reports an MAE of 4.96. The direct model, which uses only history available at the origin, scores 5.69, so 5.69 - 4.96 = 0.73 is the error the backtest hides. Run at the origin, the shift-based model scores 5.95. The origin test moves 3 of 8 shift-based columns and 0 of 6 direct columns. The seasonal-naive forecast scores 6.83.

Worked steps

  1. Hidden error: 5.69 - 4.96 = 0.73 (direct model minus shift-based backtest).
  2. Production penalty of the shift-based model: 5.95 - 5.69 = +0.26 (run at the origin minus direct).
  3. Share of the shift-based columns that read after the origin: 3 of 8.
  4. Direct model against seasonal naive: 5.69 - 6.83 = -1.14.

Use the idea

Store the forecast origin with every row, build features from history up to that origin only, keep lags of at least the horizon, and test the builder by changing values after the origin.

Where the conclusion applies

Constructed series with a stable weekday pattern and AR(1) noise, one boosting model (scikit-learn's histogram gradient boosting stands in for LightGBM) and ten panels. The size of the gap depends on how strong the short-range autocorrelation is. Production-side filling of a missing lag with the last known value is one common choice; other fills would give other deployed scores, and the origin test, not the fill, is what reveals the problem. The seasonal-naive line repeats the value from the same weekday the week before.

Check your understanding: Why does the direct model's error grow with the horizon, while the shift-based backtest stays flat?
Autocorrelation decays: the nearest known day says less about a target that is further away. The direct model can only use lags of at least h, so its information shrinks as h grows, and its error rises until only the weekly lags remain (6.03 at both 5 and 7 days). The shift-based backtest keeps reading lag 1 from the true series at every horizon, so it never loses information, and its error stays near 5.0. The gap between the two is the error the backtest hides.

Chapter 52 source: section "One-Step and Three-Step Forecasts Use Different Histories".