Winning Kaggle the Reproducible Way

Chapter 63

Lessons from the Expanded Winning-Solution Corpus

Borrow a decision with its conditions: a method's presence in winning systems documents use, not benefit.

How often a technique appears among winners is not its effect. One demonstration simulates thousands of constructed competitions and compares the share of winners using a technique with what a paired on-off comparison measures.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

How often winners use a technique, against what the technique does

When winners use a technique more often than the field does, how much does that say about the technique's effect?

Winning is a selection on score. If stronger teams are likelier to try T, winners contain extra T users whatever T does, so the share of winners using T stays well above the field's 70% even when T has no effect, or hurts. Only when adoption ignores skill does the share track the effect, and even then a high share is reached by modest effects. A paired comparison runs the same team with and without T, which cancels skill.

Equation: The apparent lift equals the share of winners using the technique minus the share of all teams using it

Scroll sideways for the whole equation

T is the technique, p_win the share of first-place teams that used T, p_field the share of all teams that used T, and the effect is the average change in a team's score caused by using T, in skill standard deviations.

Predict first. Technique T lowers every user's score by 0.5 skill standard deviations. The field adopts it at 70%, and stronger teams are somewhat likelier to adopt. What share of first-place teams used T?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: How often winners use a technique, against what the technique does. Left: share of first-place teams using technique T against its true effect, for skill-linked adoption and for adoption unrelated to skill, with the field's 70% as a dashed line; at an effect of +0.00 the shares are 93% and 70%. Right: sixty paired on-off estimates of the effect, averaging -0.017, against the true effect.
True effect of technique T on a team's score (skill standard deviations): 0: no effect
Constructed example: 2,000 simulated competitions of 40 teams, seeded, measured by the chapter activity; no real competition data.

Calculated values

Winners using T, skill-linked adoption
93%
Winners using T, unrelated adoption
70%
Field adoption
70%
Paired ablation, mean estimate
-0.017
Paired ablation, spread
0.203

With a true effect of +0.00, 93% of first-place teams used T when stronger teams adopt it more, against 70% of the whole field: 0.932 - 0.700 = 0.232 of apparent lift. When adoption ignores skill the share is 70%. The skill-linked share only falls back to the field's rate when T lowers scores by about 0.9. A paired on-off comparison averages -0.017 across 60 experiments (spread 0.203).

Worked steps

  1. Apparent lift in the skill-linked world: 0.932 - 0.700 = 0.232.
  2. Apparent lift when adoption ignores skill: 0.696 - 0.700 = -0.004.
  3. Paired ablation error: -0.017 - 0.000 = -0.017 (average estimate minus true effect).
  4. Effect at which the skill-linked share meets the field rate: -0.88.

Use the idea

Read a mention count as a record of use. Before borrowing a technique, look for a with-and-without comparison on the same setup, and reproduce it on your own validation.

Where the conclusion applies

Constructed data and one assumed link between skill and adoption (odds multiply by 2.7 per skill standard deviation); nothing here measures how real teams adopt techniques. The size of the inflation depends on that link, which is why the unrelated world is shown beside it. The paired ablation has no confounding by construction, and its estimate still varies with a standard deviation of about 0.2 across 30-team experiments.

Check your understanding: With no effect at all, 93% of winners use T. How would a reader tell this from a technique that works?
A share of 93% is what 70% adoption and skill-linked adoption produce when T does nothing, and it differs little from the 99% seen when T helps by 0.75. When adoption is unrelated to skill, the same no-effect case gives 70%. The count alone cannot separate these cases. A paired on-off comparison can: its estimate at no effect averages -0.017 with a spread of 0.203 across 60 experiments, while at an effect of 0.75 it averages 0.733.

Chapter 63 source: section "Follow the Evidence Boundary".