Winning Kaggle the Reproducible Way

Chapter 31

Deep Learning for Tabular Data

Report a neural model's standalone quality and its marginal blend contribution separately.

A neural model that loses to a credible boosted-tree baseline on its own can still improve a blend, but how much depends on the data and on how the weight is chosen. One demonstration measures the standalone and blended error as the training set grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A weaker neural model in a gradient-boosted blend, by training-set size

Does a neural model that is worse on its own lower the blend's error, and does that survive as the tree model gets more data?

The network is clearly worse than the trees at every size here, but its errors are only partly the same as the trees' errors, so a small weight on it (well under one half) beats both. Equal averaging gives the weaker model too much say: it never beats the weighted blend, and from 1,000 rows up it is worse than the trees alone. When the training set is small the trees are further from the truth and the network's smooth fit repairs more; as the trees get more data their errors shrink and become more like the network's, so the benefit shrinks. A weight chosen on a few development rows is noisy and gives some or all of the gain back.

Equation: the blend prediction is one minus w times the tree prediction plus w times the network prediction, with w chosen to minimize squared error on the development rows

Scroll sideways for the whole equation

p_g and p_n are the boosted-tree and network predictions, w the weight on the network (0 to 1) and the blend is (1 - w) p_g + w p_n. The weight minimizes squared error on development rows only.

Predict first. With 400 training rows the network's error is about 1.5 times the trees'. Which way of blending the two beats the trees alone on new rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A weaker neural model in a gradient-boosted blend, by training-set size. Left: bars of mean squared error on new rows for the trees (0.464), the network (0.682), an equal blend (0.476) and two development-weighted blends (0.452 and 0.446) with 400 training rows. Right: error removed versus the trees for each blend, with a dot per dataset.
Training rows for both models: 400
Constructed example: six seeded synthetic regression datasets, a development-tuned histogram gradient boosting model and a small scikit-learn network, measured by the chapter activity.

Calculated values

Trees alone
0.464
Network alone
0.682
Equal blend
0.476
Blend, weight from 600 rows
0.446
Blend, weight from 60 rows
0.452
Residual correlation
0.68

With 400 training rows the trees score 0.464 and the network 0.682, so the network is worse on its own. A blend with the weight chosen on 600 development rows scores 0.446, and 0.464 - 0.446 = 0.018 is the error it removes versus the trees. The chosen weight on the network averages 0.19; equal averaging scores 0.476. Choosing the weight on 60 rows scores 0.452 with weights between 0.00 and 0.35 across datasets. The two models' errors correlate at 0.68.

Worked steps

  1. Trees alone: 0.464; network alone: 0.682 (mean squared error on new rows).
  2. Blend, weight from 600 rows: 0.464 - 0.446 = +0.018 versus trees.
  3. Equal blend: 0.464 - 0.476 = -0.012 versus trees.
  4. Blend, weight from 60 rows: 0.464 - 0.452 = +0.012 versus trees.

Use the idea

Report standalone quality and the blend gain separately, on the same eligible rows and folds as a credible tree baseline. Choose the weight on development rows with nonnegative weights that sum to one, and look at both endpoints (each model alone) and equal averaging before keeping the blend.

Where the conclusion applies

Constructed data. A scikit-learn network with one small hidden layer stands in for the chapter's neural models, and a histogram gradient boosting regressor for LightGBM; real tabular networks are larger and trained on more rows, so the sizes here describe this generator only. The trees are tuned on the development rows and the network is not (it has fixed scaling and weight decay), so the comparison is deliberately tough on the network, as the chapter's credible-baseline rule asks; a better network would change the blend weights.

Check your understanding: How does the blend's advantage over the trees change from 150 to 3,000 training rows, and why?
It falls from 0.040 to 0.003 mean squared error (0.649 to 0.609 at 150 rows, 0.296 to 0.293 at 3,000). The trees' errors shrink with data and become more correlated with the network's (residual correlation 0.63 to 0.81), so there is less for the network to repair. At 3,000 rows the gain is tiny, and a weight chosen on 60 rows gives all of it back (0.296).

Chapter 31 source: section "Adding DL to a GBM Ensemble".