Winning Kaggle the Reproducible Way

Chapter 26

Advanced Target Encoding

Shrinkage shares information across categories; how much it should share depends on how much categories truly differ.

The smoothed mean from Chapter 9 and the Weight of Evidence map are both shrinkage rules with a smoothing strength. One demonstration measures log loss on new rows for a range of smoothing strengths as the true spread of category effects changes.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

How much to smooth a target encoding, by how much categories really differ

How much smoothing does a target encoding need, and does the answer change with how strongly categories really differ?

A category's raw mean is noisy when it has few rows, and the noise grows when the model trusts the column. A pseudocount pulls each mean toward the global mean in proportion to how few rows support it. When categories barely differ, strong pulling costs nothing and removes noise; when they differ strongly, strong pulling erases real effects. The best m therefore falls as the true spread rises, and no smoothing at all hurts rare categories most.

Equation: e c equals s c plus m times mu, divided by n c plus m; and the weight of evidence of category c is the log of the ratio of the probability of c among positives to the probability of c among negatives

Scroll sideways for the whole equation

e_c is the encoding of category c, s_c the sum and n_c the count of its training labels, mu the global mean and m the pseudocount; WoE(c) is the log ratio of the category's share among positives to its share among negatives.

Predict first. With a moderate true spread of 1.0, how does an encoding with no smoothing (m = 0) compare with the Chapter 9 default m = 10 on log loss for new rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: How much to smooth a target encoding, by how much categories really differ. Two curves of log loss against pseudocount at category spread 1.0: all new rows on the left, rare categories on the right, each with a circle at its lowest point and a dashed line for weight of evidence; the lowest overall is at m = 10.
True spread of category effects (standard deviation, log odds): 1.0
Constructed example: thirty seeded synthetic tasks per setting with Zipf-like category frequencies and a logistic regression, measured by the chapter activity.

Calculated values

Best pseudocount m
10
Log loss at best m
0.580
Log loss with no smoothing
0.619
Log loss at m = 10
0.580
Weight of evidence
0.584

With category spread 1.0, the lowest new-row log loss is 0.580 at m = 10. No smoothing scores 0.619, so 0.619 - 0.580 = 0.039 is what skipping smoothing costs; the Chapter 9 default m = 10 is the best setting here. On rare categories (6.7% of new rows) no smoothing scores 0.695 against 0.621 at m = 10. Weight of evidence at smoothing 0.5 scores 0.584.

Worked steps

  1. Best pseudocount: m = 10 with log loss 0.580.
  2. No smoothing: 0.619 - 0.580 = 0.039 worse than the best.
  3. Default m = 10: 0.580 - 0.580 = 0.000 worse than the best.
  4. Rare categories: m = 0 gives 0.695, m = 10 gives 0.621; weight of evidence 0.640.

Use the idea

Keep Chapter 9's smoothed mean as the baseline, choose m by validation inside the outer boundary rather than by habit, and judge any richer encoder by whether it beats the best smoothed mean on rows it never saw, with rare categories reported separately.

Where the conclusion applies

Constructed data with normally distributed category effects, a logistic model and cross-fitted columns. Weight of Evidence is the chapter's fit_woe formula with smoothing 0.5. No James-Stein, mixed-model or native-categorical encoder is run, so this does not rank those. When categories barely differ the differences between m values are within about 0.002 log loss and should not be over-read.

Check your understanding: Why does the best pseudocount fall as the true spread of category effects grows, and why does skipping smoothing hurt rare categories most?
Smoothing trades noise for bias. When categories barely differ there is little real effect to erase, so a large m (30 or more) removes noise at no cost; when they differ strongly (spread 1.5) the best m is 3, because heavy pulling would flatten real differences. A rare category's own mean rests on a handful of rows, so with m = 0 the model trusts noise (rare-category log loss 0.750 at spread 1.5 against 0.588 at m = 3).

Chapter 26 source: section "Which One to Use When".