Winning Kaggle the Reproducible Way

Chapter 34

Pseudo-Labeling

A teacher is eligible by the labels it trained on, not by its fold number.

Pseudo-labels are only as clean as the labels their teacher saw. One demonstration compares the eligible teacher with the common shortcut, averaging every fold model except k, and measures how far each pipeline's cross-validation score sits from its score on fresh rows as the number of recipients grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

The eligible teacher against the "all models except k" teacher, by number of recipients

How much does the shortcut teacher inflate the student's cross-validation score, and what does honest pseudo-labeling add?

Model k never saw fold k, but the other models did. Averaging them gives pseudo-labels that carry what fold k's labels taught those models, so the student arrives at fold k partly informed about it and cross-validation rewards the leak. The more recipients, the more of that information reaches the student. The eligible teacher's pseudo-labels carry only outer-training information, so the estimate stays honest and the gain is small.

Equation: the eligible teacher probability for fold k is the output of model k; the shortcut teacher probability is the average of the outputs of all other models j different from k

Scroll sideways for the whole equation

m_j is the model that excludes fold j, x a recipient, p_k(x) the teacher probability used for outer fold k, and K the number of folds. The eligible teacher is m_k; the shortcut averages m_j over j not equal to k, and each of those models trained on fold k's labels.

Predict first. With 1,500 recipients, the shortcut teacher's cross-validation AUC is how far above the same student's score on fresh rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: The eligible teacher against the "all models except k" teacher, by number of recipients. Paired bars of cross-validation AUC and fresh-row AUC for the supervised baseline, the eligible teacher and the all-models-except-k teacher at 1,500 recipients; the shortcut reports 0.054 above its fresh-row score.
Unlabeled recipients available for pseudo-labeling: 1,500
Constructed example: eight seeded synthetic binary tasks, fold models as teachers and a histogram gradient boosting student, measured by the chapter activity.

Calculated values

Shortcut CV
0.775
Shortcut, fresh rows
0.721
Eligible CV
0.707
Eligible, fresh rows
0.709
Baseline CV
0.704
Baseline, fresh rows
0.705

With 1,500 recipients the shortcut teacher reports 0.775 but its student scores 0.721 on fresh rows, 0.775 - 0.721 = 0.054 of optimism, a claimed gain of 0.071 over the baseline's 0.704. The eligible teacher reports 0.707 and scores 0.709 on fresh rows, -0.002 apart, a real change of +0.004 against the baseline's 0.705. The teachers kept 514 (eligible) and 432 (shortcut) pseudo-labels per fold, with accuracy 0.758 and 0.794.

Worked steps

  1. Shortcut: 0.775 - 0.721 = 0.054 (estimate minus fresh rows).
  2. Eligible: 0.707 - 0.709 = -0.002.
  3. Claimed gain of the shortcut: 0.775 - 0.704 = 0.071.
  4. Real gain of the eligible teacher on fresh rows: 0.709 - 0.705 = +0.004.

Use the idea

For every outer fold build teachers from outer-training labels only (a fold model, an inner-fold ensemble), add recipients to training only, and score untouched genuine labels. Report the supervised baseline on the same folds, and audit the pseudo-labels' accuracy on any rows where the truth is known.

Where the conclusion applies

Constructed data with a stable relationship and a histogram gradient boosting model for teacher and student. The shortcut student is also better on fresh rows than the baseline here, because its teachers together saw all 600 labels; that real gain is small and roughly what a fit on all 600 labels delivers. What the shortcut inflates is the cross-validation estimate. A single run of 3 folds is noisy, so eight datasets are averaged.

Check your understanding: What does honest pseudo-labeling add over the supervised baseline at 1,500 recipients, and what did the shortcut claim?
The eligible teacher improves the fresh-row score from 0.705 to 0.709: a gain of 0.004, small but genuine. The shortcut claims 0.775 against the baseline's 0.704, a gain of 0.071, while its fresh-row score is 0.721, 0.016 above the baseline.

Chapter 34 source: section "Trace the Teacher's Training Labels".