Winning Kaggle the Reproducible Way

Chapter 36

Categorical Embeddings

Every categorical encoding needs a rule for categories it has never seen.

Test rows often carry categories the training dictionary lacks. One demonstration measures how a label code, a frequency encoding and a nested target encoding with two different unknown rules hold up as the share of unseen-category rows grows, against a model with no categorical feature at all.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Encodings against test rows from unseen categories

How large can the share of unseen categories get before an integer label code is worse than dropping the column, and what does the unknown rule cost a target encoding?

Numbered in order of first appearance in a shuffled table, label codes follow popularity, so the model's splits on them partly track how common a category is. The unknown code -1 sits beside the codes of the most popular categories, so every row from a new category receives their low prediction, although new categories are rare and run hotter here. Once unseen rows are common, that misplaced unknown costs more than the seen rows gain. Numbered in random order, the same code stays above the no-feature baseline. A frequency count has a natural value for a new category (zero occurrences), which is informative when rarity matters. The smoothed target mean returns exactly the prior when n_c = 0, which is the documented fallback; writing 0 instead tells the model the category almost never has events. The seen rows are unaffected by either unknown rule.

Equation: e c equals s c plus m times mu, divided by n c plus m; when n c is zero e c equals mu

Scroll sideways for the whole equation

e_c is the encoding of category c, s_c the sum and n_c the count of permitted training targets in c, mu their mean (the prior) and m the smoothing strength (10 here). For an unseen category n_c = 0.

Predict first. Half the test rows come from categories absent in training. How does the integer label code (unknown = -1) compare with a model that has no categorical feature at all?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Encodings against test rows from unseen categories. Left: pooled AUC of five encodings when 25% of test rows are unseen; the label code scores 0.613 and the random-order label code 0.718 against a no-feature line at 0.646. Right: AUC within seen and unseen rows for each encoding.
Share of test rows from categories absent in training: 25%
Constructed example: seeded synthetic categories and gradient-boosted models, measured by the chapter activity.

Calculated values

No categorical feature
0.646
Label code
0.613
Frequency count
0.758
Target mean, unknown = prior
0.764
Target mean, unknown = 0
0.696
Unseen rows, label code
0.607
Label code, random order
0.718

With 25% of test rows from unseen categories (measured 27%), the label code scores 0.613 against 0.646 with no categorical feature, 0.613 - 0.646 = -0.033: worse than dropping the column. The target mean with the prior scores 0.764; with unknown = 0 it scores 0.696, so the unknown rule alone costs 0.764 - 0.696 = 0.068 AUC. Seen rows score 0.770 under both target rules. Best overall here: target mean with the prior, with frequency alone 0.006 behind the best target rule. Numbered in random order, the label code scores 0.718, so its result depends on where -1 lands.

Worked steps

  1. Label code against no feature: 0.613 - 0.646 = -0.033.
  2. Cost of unknown = 0: 0.764 - 0.696 = 0.068.
  3. Frequency against the best target rule: 0.764 - 0.758 = 0.006.
  4. Label code fell below the no-feature model in 83% of 6 worlds.

Use the idea

Persist the training dictionary, choose the unknown value on purpose (the training prior for a target mean, zero for counts), and score the encoding on a validation slice that contains unseen categories in the share the test set will.

Where the conclusion applies

Constructed data and HistGradientBoostingClassifier standing in for a GBM library. The unseen categories are rare by construction, so frequency carries signal about them; with no rarity effect the gap between frequency and target encoding would change. New categories run hotter than average here, which enlarges the cost of any rule that places them at the low end (unknown = 0 for a target mean, or -1 beside the popular codes). The label code's unknown value is -1, a documented code with no training rows; whether the label code falls below the no-feature baseline depends on the code order, and the crossover share moves with the random draw.

Check your understanding: Filling unseen categories with 0 instead of the training prior leaves the seen-row score unchanged. Why does the pooled score still fall?
Only unseen rows receive the unknown value. With 0 they look like categories that almost never have events, so they are ranked below the seen rows although their true event rate is higher; the prior places them at an average value. At 50% unseen the pooled AUC is 0.742 with the prior rule and 0.665 with 0, while the seen-row AUC is identical (0.776).

Chapter 36 source: section "Handling Unseen Categories at Inference".