Winning Kaggle the Reproducible Way

Chapter 20

Horse Health: Small Data, Big Discipline

Decision-rule factors fitted on OOF predictions need rows they were not fitted on.

Per-class factors tuned on out-of-fold probabilities raise the accuracy of the same rows. One demonstration measures how much of that gain survives on new rows, as the number of development rows grows and for two kinds of probabilities.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Class factors tuned on OOF probabilities, scored on the same rows and on new rows

How much of the accuracy a tuned class factor adds on its own OOF rows survives on new rows?

Plain argmax is already the accuracy-optimal rule when the probabilities are not distorted, so no factor can help in expectation and the search only fits noise: it gains on its own rows and loses a little, or nothing, on new ones. When the probabilities are distorted, a factor can undo the distortion and the gain is real once there are enough rows to find it (from about 500 here). The same-row gain is positive in both cases, so it cannot tell them apart. From 500 rows up the cross-half and new-row gains both separate them; at 250 rows neither does reliably.

Equation: the predicted class is the class c that maximizes the predicted probability p c divided by its factor f c, with the factor for class zero fixed at one

Scroll sideways for the whole equation

p_c is the predicted probability of class c, f_c the factor for class c (f_0 = 1) and the predicted class is the c that maximizes p_c / f_c. Gain is accuracy with factors minus accuracy of plain argmax.

Predict first. With 988 development rows and the model's own probabilities, factors tuned on the OOF rows raise accuracy on those rows. What do they do to accuracy on new rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Class factors tuned on OOF probabilities, scored on the same rows and on new rows. Two panels of three bars each, accuracy gain of tuned class factors in percentage points on the same rows, in a cross-half check and on new rows, with 988 development rows. For the model's own probabilities the new-row gain is -0.52 points.
Development rows: 988: the chapter's development size
Constructed example: seeded synthetic three-class datasets and a boosted-tree classifier, measured by the chapter activity.

Calculated values

Plain, same-row gain (points)
1.03
Plain, new-row gain (points)
-0.52
Rebalanced, same-row gain (points)
2.06
Rebalanced, new-row gain (points)
+0.72
Plain argmax accuracy on OOF rows
0.629

With 988 development rows the model's own probabilities gain 1.03 accuracy points from tuned factors on the rows that tuned them and -0.52 on 4,000 new rows, so 1.03 - (-0.52) = 1.55 points is overstatement. For the rebalanced probabilities the same-row gain is 2.06 and the new-row gain +0.72 points: the factors pay there. The same-row gain is positive in both cases and cannot tell them apart. The cross-half gains are -0.94 and +0.79, which does separate them.

Worked steps

  1. Plain: same-row +1.03, cross-half -0.94, new rows -0.52 points.
  2. Rebalanced: same-row +2.06, cross-half +0.79, new rows +0.72 points.
  3. Plain overstatement: 1.03 - (-0.52) = 1.55 points.
  4. Rebalanced overstatement: 2.06 - 0.72 = 1.34 points.

Use the idea

Compare plain argmax first, tune factors only inside an outer assessment or on untouched rows, and keep them only if the assessed gain is positive.

Where the conclusion applies

Constructed data and one boosted-tree stand-in for the chapter's gradient boosting libraries. The rebalanced probabilities are a constructed distortion, not a recipe. Fourteen datasets per size give new-row gains with standard errors of at most about 0.4 points, so smaller differences, including the rebalanced gain at 250 rows, are not resolved.

Check your understanding: Why do the same factors help the rebalanced probabilities at 988 rows but not the model's own?
A factor can only repair a distortion. Rebalancing moves the decision boundary away from the accuracy-optimal one, and tuned factors move it back (+0.72 points on new rows at 988 rows). The model's own probabilities leave nothing to repair, so the same search only fits noise (-0.52 points). Only rows the factors were not tuned on distinguish the two cases.

Chapter 20 source: section "Per-Class Threshold Optimization on OOF".