Winning Kaggle the Reproducible Way

Chapter 57

PatchTST for Time Series and HAR

Compare patching with a feature baseline at matched forward origins, and choose patch length from the phenomena.

Patching turns short blocks into tokens, and the patch length decides what a token can see. One demonstration compares a patch-token stand-in with a feature baseline, the raw window and a seasonal-naive forecast on the same forward origins, and counts the tokens that each patch length creates.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A patch-token stand-in against a feature baseline, by patch length

Does patching beat a feature baseline at the same forward origins, and how does the patch length change the answer and the token count?

Short patches keep the local shape of a pulse, so the embedded tokens expose the oscillation that predicts the rebound. Long patches average that shape away inside one token, and with few tokens the model sees little more than a coarse summary, which the feature baseline also provides. Shorter patches cost more tokens, and attention cost grows with the square of the token count.

Equation: the number of tokens equals the floor of the quantity T minus P, divided by S, plus 1

Scroll sideways for the whole equation

T is the window length (96), P the patch length, S the stride (P/2) and the number of tokens without padding is floor((T - P)/S) + 1. MAE is the mean absolute forecast error, so lower is better.

Predict first. With a patch length of 24 steps, how does the patch-token model compare with the feature baseline?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A patch-token stand-in against a feature baseline, by patch length. Bars of forecast error with patch length 12: seasonal naive 0.653, feature baseline 0.447, raw window 0.409, patch tokens 0.414, with a dot for each of six sets of series.
Patch length P in steps (stride is P/2; the pulse oscillation lasts 10): 12: 15 tokens
Constructed example: six seeded sets of 30 synthetic series, a patch-token stand-in with gradient boosting, measured by the chapter activity.

Calculated values

Patch-token MAE
0.414
Feature baseline MAE
0.447
Raw window MAE
0.409
Seasonal naive MAE
0.653
Tokens per window
15

With patches of 12 steps (stride 6) each 96-step window becomes 15 tokens, floor((96 - 12) / 6) + 1 = 15. The patch model scores an MAE of 0.414 and beats the feature baseline at 0.447, so 0.447 - 0.414 = 0.033 separates them. The raw 96-step window scores 0.409 and the seasonal-naive forecast 0.653. Attention would relate 225 pairs of tokens per window.

Worked steps

  1. Tokens: floor((96 - 12) / 6) + 1 = 15.
  2. Feature baseline minus patch model: 0.447 - 0.414 = +0.033 (positive means patching is better).
  3. Raw window minus patch model: 0.409 - 0.414 = -0.005.
  4. Token pairs for attention: 15 x 15 = 225.

Use the idea

Compare a patch model with a feature baseline and the raw window on identical forward origins, pick the patch length from the duration of the events that matter, and check the token count and cost before a long run.

Where the conclusion applies

A stand-in, not PatchTST: patches share a PCA projection and feed gradient boosting, with no positional encoding, attention or end-to-end training, because the neural library is not installed here. The rebound makes the target depend on local pulse shape by construction, which favours a representation that sees shape; a signal that is a pure cycle gives the patch and feature models no such gap. One window length, one horizon and six sets; the raw-window comparator already uses all 96 values.

Check your understanding: A patch length of 4 scores best here. Why is that not simply the answer to use?
P = 4 creates 47 tokens against 15 for P = 12, about ten times as many token pairs for attention to relate, and its error gain is 0.025 (0.389 against 0.414). Whether that is worth the cost depends on the budget; the chapter asks for the actual token count and cost to be measured alongside the score. It also depends on the phenomena: the pulses here last about 10 steps.

Chapter 57 source: section "PatchTST vs. Feature-Engineered GBM: When to Use Each".