Winning Kaggle the Reproducible Way

Chapter 28

Nearest-Neighbor Features

A neighbour target summary is a label-derived feature: exclude the row itself and build it inside the fold.

A feature built from the labels of the K nearest training rows leaks the row's own label if the row is its own neighbour. One demonstration builds the feature three ways and measures how far each cross-validation estimate sits from the score on new rows, for K from 1 to 50.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A neighbour target-mean feature, scored by cross-validation and on new rows

How many neighbours does it take before including the row itself stops distorting the estimate, and what does the legitimate feature gain?

A row stored in its own database is its own nearest neighbour at distance zero, so its label supplies 1/K of the feature. At K = 1 the feature is the label. A tree model finds that immediately: cross-validation reads near-perfect, but new rows have no such column and the deployed model leans on it. As K grows the row's own share shrinks and the distortion fades. Excluding the row once on the full set already removes most of it; nesting also keeps the validation labels out of the training rows' features.

Equation: f i equals one over K times the sum of y j over the K nearest database rows j of row i

Scroll sideways for the whole equation

f_i is the neighbour feature of row i, N_K(i) the K database rows nearest to it, y_j the label of row j and K the number of neighbours. If row i is in its own database, its label y_i is one of the K terms and carries weight 1/K.

Predict first. With K = 1 and the row included among its own neighbours, what does cross-validation report, and what does the model score on new rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A neighbour target-mean feature, scored by cross-validation and on new rows. Paired bars of cross-validation AUC and new-row AUC for three neighbour-feature constructions at K = 5, with a dashed line at the no-feature score 0.836; the self-included estimate is 0.065 above its new-row score.
Neighbours K in the target-mean feature: 5
Constructed example: six seeded synthetic datasets with clustered labels and a histogram gradient boosting model, measured by the chapter activity.

Calculated values

Self-included CV
0.921
Self-included, new rows
0.856
Nested CV
0.853
Honest feature, new rows
0.858
No feature, new rows
0.836

With K = 5: the self-included pipeline reports 0.921 but scores 0.856 on new rows, 0.921 - 0.856 = 0.065 of optimism. Excluding the row once reports 0.853; the nested estimate 0.853 differs from the new-row score 0.858 by -0.005. Against 0.836 without the feature, the honest feature adds 0.022 AUC on new rows. In a self-included feature the row's own label is 1/5 of the mean.

Worked steps

  1. Self included: 0.921 - 0.856 = 0.065 (estimate minus new rows).
  2. Self excluded, then CV: 0.853 - 0.858 = -0.005.
  3. Nested: 0.853 - 0.858 = -0.005.
  4. Value of the honest feature on new rows: 0.858 - 0.836 = +0.022.

Use the idea

Build neighbour target features like a target encoding: exclude the row itself, cross-fit inside each outer training fold, query validation rows from the outer-training database only, and keep a no-feature baseline on matched folds.

Where the conclusion applies

Constructed data with strong local structure, so the legitimate gain is real here and may be smaller on a competition table. In this generator excluding the row once on the full training set and nesting give estimates within about 0.01 AUC of each other and of the new-row score; with other data the second boundary can matter more, which is why the chapter nests by default. A histogram gradient boosting classifier stands in for LightGBM.

Check your understanding: At K = 5, how much does the self-included estimate overstate the score on new rows, and what does the honest feature add?
The self-included estimate is 0.921 against 0.856 on new rows, an overstatement of 0.065, while the nested estimate (0.853) is within 0.01 of the new-row score (0.858). Compared with 0.836 for the model without the feature, the honest feature adds 0.022 AUC here. The leak made it look like 0.085.

Chapter 28 source: section "A Three-Row Query".