Winning Kaggle the Reproducible Way

Chapter 16

The Fourth-Root Blend

Equal averaging is the comparator; a blend must beat the strongest model on rows that chose none of its weights.

Blending pays only when the models' errors differ. One demonstration measures equal, fourth-root and development-fitted blends against the best single model as the models' prediction correlation rises.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Blending three similar models as their errors become alike

How alike can three models' predictions be before a blend stops beating the best single model, and do the weights matter?

Averaging cancels each model's own noise but not the noise the models share. With independent errors the blend removes a lot; as the shared part grows the blend's gain shrinks, and because the three models are not equally good the average eventually does worse than simply keeping the best one. Fourth-root weights compress the margins so far that they land close to equal weights.

Equation: w i equals the fourth root of the larger of m i and zero, divided by the sum over j of the same quantity, where m i is the baseline loss b minus model i's loss

Scroll sideways for the whole equation

m_i is model i's margin over the baseline (baseline loss minus its development loss), b the loss of a defined dummy predictor and w_i the weight of model i. Gain over the best model is its assessment log loss minus the blend's; positive means the blend wins.

Predict first. When the three models' errors are almost identical (shared noise 0.95), how does the equal-weight blend compare with the best single model?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Blending three similar models as their errors become alike. Left: log loss gain over the best single model for equal (-0.010), fourth-root (-0.009) and fitted (+0.000) weights at prediction correlation 0.97. Right: mean weight of each of the three models under fourth-root, equal and fitted weights; fitted weights give the least noisy model 0.99.
Share of each model's noise that is common to all three: 0.95: nearly identical errors
Constructed example: seeded synthetic data and three logistic regressions, measured by the chapter activity.

Calculated values

Prediction correlation
0.97
Best single model
0.613
Equal blend
0.623
Fourth-root blend
0.623
Fitted blend
0.613

At prediction correlation 0.97, the equal blend does worse than the best single model: 0.613 - 0.623 = -0.010 log loss on 20,000 assessment rows, and it beats the best model in 0% of 40 datasets. Fourth-root weights score 0.623 (weights differ by 0.028 on average). Against equal weights, fitted weights change log loss by -0.0100 on the rows the weights were fitted to and by -0.0096 on assessment rows (negative means fitted weights do better).

Worked steps

  1. Equal blend: 0.613 - 0.623 = -0.010 (best model minus blend; positive means the blend wins).
  2. Fourth-root blend: 0.613 - 0.623 = -0.010.
  3. Fitted blend: 0.613 - 0.613 = 0.000.
  4. Fourth-root weights differ from each other by 0.028 on average (equal weights differ by 0).

Use the idea

Check prediction correlation first as a screen, then compare equal weights, the fourth-root heuristic and any fitted weights with the strongest single model on assessment rows. Keep the single model when the blend does not beat it.

Where the conclusion applies

Constructed data and three logistic regressions standing in for three different model families; the models differ only in noise, whereas real models also differ in bias. The baseline for the fourth-root rule is the measured log loss of a constant-prior predictor, as the chapter requires, not a target mean passed off as a score. Correlation here is measured on the 300 development rows.

Check your understanding: Fourth-root weights and equal weights score almost the same here. Why, and does that make the heuristic a good idea?
The fourth root compresses margins so far that the three weights differ by only 0.028 on average, so the blend is nearly an equal average (log loss 0.623 for both). That is the chapter's point: it is a mild heuristic, not an optimality claim. It earns its place only if it beats equal weights on assessment rows, and here the difference is within noise.

Chapter 16 source: section "The Fourth-Root Weight Formula".