Winning Kaggle the Reproducible Way

Chapter 24

Building a Kaggle Portfolio

Agree on a prediction-artifact contract (IDs, folds, order) before stacking a teammate's model.

A teammate's out-of-fold file that arrives without IDs can be joined by position and silently misalign. One demonstration measures what the blend, the teammate's model and a stacked blend look like as more of the file's rows are out of order.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A teammate's OOF file joined by position or by ID

How much of a blend's gain is lost, and what does it look like, when a teammate's OOF file is joined by position but is partly out of order?

Both files still have one finite probability per position, so nothing fails to run. A join by position pairs some of model A's predictions with predictions for other rows, so teammate B looks weaker than it is and both an average blend and a stack gain less; once the file is fully shuffled the stack learns to ignore the column. A join by ID restores every pair, and the full blend gain.

Equation: the blended probability for position i is the mean of A's prediction a i and B's prediction at the row the file holds in that position, which is row i only when joined by ID

Scroll sideways for the whole equation

a_i and b_i are the two models' out-of-fold probabilities for row i, pi(i) the row that the file holds at position i, and the blend is the mean of the two columns.

Predict first. A quarter of teammate B's rows are saved out of order and the blend is joined by position. Where does its AUC land compared with the join by ID and with model A alone?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A teammate's OOF file joined by position or by ID. Bars of AUC for model A alone, teammate B in file order, the blend by ID, the average blend by position and the stack by position with 25% of rows out of order, and a dashed line at model A alone, 0.771.
Share of the teammate's rows saved out of order: 25%
Constructed example: six seeded synthetic tasks, logistic regression and extremely randomized trees, measured by the chapter activity.

Calculated values

Blend by ID
0.813
Average blend by position
0.798
Model A alone
0.771
Teammate B alone, file order
0.622
Positions matching their IDs
75%

With 500 of 2000 rows out of order (25%), the blend joined by ID scores 0.813, model A alone 0.771, and the blend joined by position 0.798: 0.813 - 0.798 = 0.015 of AUC lost, 35% of the 0.043 the blend gains over model A. The teammate's model scores 0.622 alone in file order against 0.660 by ID, and the stacked blend by position scores 0.798 against 0.818 by ID. Only 75% of positions match their IDs, which an ID check would have reported.

Worked steps

  1. Blend by ID: 0.813; model A alone: 0.771; gain 0.813 - 0.771 = 0.043.
  2. Average blend by position: 0.798; lost 0.813 - 0.798 = 0.015.
  3. Stack by position: 0.798 against 0.818 by ID.
  4. Teammate B alone: 0.622 in file order, 0.660 by ID.

Use the idea

Agree on one saved format with an ID column, a fold column and class order; assert unique IDs and equal ID sets before every join, and join by ID, never by position.

Where the conclusion applies

Constructed data with one linear and one tree model on disjoint features, a deliberately complementary pair; real pairs gain less. Misalignment here is a random shuffle of the stated share of rows. This is a data-contract effect only and says nothing about portfolios, teams or hiring.

Check your understanding: With the file fully shuffled, the average blend falls below model A alone while the stacked blend only matches it. Why do they differ?
An average gives the shuffled column the same weight as the good one, so noise is added (0.750 against 0.771 for A alone). A stack is fitted on labels, finds that the misaligned column predicts nothing and gives it near-zero weight, so it recovers model A's score (0.770) and quietly throws away the teammate. Neither failure raises an error; a one-line ID check does.

Chapter 24 source: section "Solo or Team".