Winning Kaggle the Reproducible Way

Chapter 60

Multi-Target Learning

Compare the primary metric with and without the auxiliary task, and let relatedness and weight decide.

An auxiliary target feeds gradients into a shared layer. One demonstration measures held-out accuracy on a scarce main target, with and without a dense auxiliary target, as the auxiliary target's relatedness and loss weight change.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Held-out main-target accuracy with and without an auxiliary task

When does a dense auxiliary target improve a scarce main target on new rows, and how much does the loss weight matter?

The auxiliary gradient reaches the shared layer, so it shapes the features the main head reads. When the auxiliary target is built on the same latent features, its 1,200 dense rows teach the shared layer features that 120 main labels cannot, and a larger weight helps more. As its signal drifts away from the main target's, those extra rows teach features the main head does not need, and the gain shrinks toward noise. The weight multiplies whatever the auxiliary task teaches, useful or not.

Equation: L equals the main task mean squared error plus w times the auxiliary task mean squared error

Scroll sideways for the whole equation

L is the training loss, MSE_main the mean squared error of the main head on its labelled rows, MSE_aux the mean squared error of the auxiliary head on its rows and w the auxiliary loss weight. w = 0 is the single-task model.

Predict first. With an auxiliary target built entirely on the main target's latent features (relatedness 1.0) and loss weight 3, what happens to the main target's R-squared on new rows compared with the single-task model?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Held-out main-target accuracy with and without an auxiliary task. Left: bars of held-out R-squared for the single-task model (0.574) and for weights 0.3, 1 and 3 (0.594, 0.604, 0.657) at relatedness 0.7, with a dot per replicate. Right: the paired gain over the single-task model for each weight, with mean and two standard errors; the weight-3 gain is +0.083.
Relatedness of the auxiliary target (share of its signal built on the main target's latent features): 0.7
Constructed example: seeded synthetic regression problems and a small numpy shared-layer network, measured by the chapter activity.

Calculated values

Single-task R-squared
0.574
R-squared at weight 0.3
0.594
R-squared at weight 1
0.604
R-squared at weight 3
0.657
Gain at weight 3
+0.083
Gain at weight 3, in standard errors
4.7

At relatedness 0.7 the single-task model scores R-squared 0.574 on new rows. With the auxiliary task at weight 3 it scores 0.657, and 0.657 - 0.574 = 0.083 (standard error 0.018, better in 92% of 12 replicates). The gain is more than two standard errors, so the auxiliary task earns its place. Weight 1 changes the score by +0.030 and weight 0.3 by +0.020.

Worked steps

  1. Single-task model: R-squared 0.574 on 4,000 new rows.
  2. Weight 3: 0.657 - 0.574 = 0.083, with standard error 0.018.
  3. Weight 1: 0.604 - 0.574 = 0.030.
  4. Weight 0.3: 0.594 - 0.574 = 0.020.

Use the idea

Train the multi-target model and a single-task reference under the same stopping rule, compare the primary metric on rows neither used, and keep the auxiliary task only if the gain is clear, then tune its weight.

Where the conclusion applies

Constructed data and a small numpy network standing in for the chapter's PyTorch trunk. The main-target stopping step comes from only 30 rows, which is noisy, and the size of the gain differs between problems, so the chart shows each replicate. No gradient alignment is measured: a small or negative change is not proof of gradient conflict, which is the chapter's own caution. Under two other seeds the weight-3 gain at relatedness 1.0 stayed clear (about 0.07) and the related settings kept beating the unrelated ones, but the weight-3 gain at 0.7 fell to about 0.03, so the exact sizes and the order of 1.0 and 0.7 are not stable.

Check your understanding: At relatedness 0, is the weight-3 model better than the single-task model, and what does that say about adding an auxiliary task by default?
At relatedness 0 the weight-3 model changes R-squared by -0.004, against a standard error of 0.025, so there is no evidence of any gain, and the weight-1 model is 0.002 below the single-task model. At relatedness 1.0 the weight-3 gain is 0.079, about 5.8 standard errors. The auxiliary task has to earn its place against the single-task reference on the primary metric.

Chapter 60 source: section "Auxiliary Tasks: What Qualifies".