Winning Kaggle the Reproducible Way

Chapter 12

The GBM Trio

Compare configurations on common rows and a stated budget; a framework name settles nothing.

The chapter treats the framework as a bundle of choices: how a category is represented, how rounds are chosen, what it costs. One demonstration holds the boosted-tree budget fixed and measures how three category representations compare as the number of levels grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Three ways to feed a category to a boosted-tree learner

How much does it matter whether a category is fed as ordered integer codes, as native categorical splits or as one-hot columns?

With few levels, the boosted trees recover every level whichever way it is fed. As the number of levels grows, ordered codes force the trees to carve the level effects out of one integer axis, and the score falls behind. Native categorical splits group levels by their effect directly. One-hot columns recover most of the loss at the price of one column per level.

Equation: Delta for a representation equals AUC with native splits minus AUC with that representation

Scroll sideways for the whole equation

AUC_rep is the AUC on new rows when the category is represented as rep (ordinal, native or onehot), and the gap is measured against the native representation.

Predict first. With 128 levels, how much AUC does treating the integer codes as ordered numbers cost compared with native categorical splits?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Three ways to feed a category to a boosted-tree learner. Bars of AUC on new rows for a category with 128 levels: no category 0.707, ordered codes 0.751, native splits 0.782 and one-hot 0.769, with dots for each of five datasets.
Levels in the category: 128 (about 23 rows per level)
Constructed example: seeded synthetic data and scikit-learn histogram gradient boosting, measured by the chapter activity.

Calculated values

Native splits
0.782
Ordered codes
0.751
One-hot
0.769
No category
0.707
One-hot columns
132

With 128 levels (about 23 training rows each), native categorical splits score 0.782 AUC. Ordered integer codes score 0.751, so 0.782 - 0.751 = 0.031. One-hot columns score 0.769 with 132 columns instead of 5. Leaving the category out scores 0.707. The choice of representation is worth a comparison at this size.

Worked steps

  1. Ordered codes behind native: 0.782 - 0.751 = 0.031 AUC.
  2. One-hot behind native: 0.782 - 0.769 = 0.013 AUC.
  3. Value of the category: 0.782 - 0.707 = 0.075 AUC.
  4. Columns fed to the trees: 5 (codes, native) against 132 (one-hot).

Use the idea

Run the representations your library supports on the same folds with the same rounds and leaves, and keep the cheapest one that is within noise of the best. For a high-cardinality column, include a native or target-aware path in the comparison.

Where the conclusion applies

Constructed data and scikit-learn's HistGradientBoostingClassifier as the stand-in: LightGBM, XGBoost and CatBoost are not installed here. The three representations correspond to treating a label-encoded column as numeric, to the native categorical paths of LightGBM and CatBoost (and XGBoost with categorical support enabled), and to one-hot encoding. The numbers belong to this learner and generator. Fit time is not reported because it varies by machine; the column count is the deterministic cost proxy.

Check your understanding: Why is the representation choice nearly irrelevant at 8 levels, and what does one-hot buy at 128 levels?
With 8 levels each level has hundreds of rows and a few splits on the code isolate it, so every representation reaches the same score (0.791 for ordinal against 0.792 for native). At 128 levels there are about 23 rows per level and ordered codes need many splits on one integer axis, so they fall to 0.751 against 0.782 for native. One-hot recovers most of that (0.769) but feeds the trees 132 columns. Native splits stayed ahead of one-hot in each of the five datasets at 128 levels, while up to 64 levels the two were within 0.01, so that ordering belongs to this learner and generator rather than to a general ranking. The mechanism is the pattern measured here, not a proof.

Chapter 12 source: section "Feature Types and Framework Fit".