Winning Kaggle the Reproducible Way

Chapter 11

When Feature Engineering Fails

Diagnose the representation, boundary or selection dependency before changing the algorithm.

PCA keeps the directions with the most variance, which need not be the directions that carry the label. One demonstration keeps k components of a constructed embedding and measures variance kept against AUC on new rows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

How much variance PCA keeps, and how much label it keeps

How many principal components does a reduction need before it keeps the label, and what does the variance kept say about that?

Components are ordered by variance, not by use. When the label lives in weak directions, the first eight components capture almost all the variance and none of the label, and the model scores about chance. Once k reaches the label's variance rank the score jumps. When the label sits in the strongest directions, PCA is fine from the start. The same reduction fails in one world and works in the other.

Equation: V of k equals the sum of lambda j for j from 1 to k, divided by the sum of lambda j for j from 1 to 64

Scroll sideways for the whole equation

lambda_j is the variance of the j-th principal component, k the number of components kept, and V(k) the share of total variance those k components keep.

Predict first. With 8 principal components, which keep about 93% of the variance, what AUC does the model reach when the label sits in weaker directions?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: How much variance PCA keeps, and how much label it keeps. Left: AUC against variance kept for PCA with the label in weaker or strongest directions, with the unreduced model as a dashed line. Right: AUC at 8 components, 0.507 in world A and 0.818 in world B, against the unreduced 0.784 and LDA 0.792.
Principal components kept (k): 8: the strong directions
Constructed example: seeded synthetic embeddings, PCA, LDA and logistic regression, measured by the chapter activity.

Calculated values

Components kept
8
Variance kept (world A)
93.3%
PCA, label in weaker directions
0.507
PCA, label in strongest directions
0.818
Unreduced, 64 dimensions
0.784

With 8 components the reduction keeps 93.3% of the variance in world A (95.0% in world B). When the label sits in weaker directions it scores 0.507 AUC on new rows, which is no better than chance; the unreduced model scores 0.784, so 0.507 - 0.784 = -0.277. When the label sits in the strongest directions the same reduction scores 0.818. A shrinkage LDA fitted inside training scores 0.792 without choosing k.

Worked steps

  1. Variance kept by 8 components, world A: 93.3%.
  2. World A, PCA minus unreduced: 0.507 - 0.784 = -0.277 AUC.
  3. World B, PCA minus unreduced: 0.818 - 0.782 = 0.036 AUC.
  4. World A, LDA minus PCA: 0.792 - 0.507 = 0.285 AUC.

Use the idea

Compare the unreduced representation with fold-fitted PCA at several k, scored on the same assessment. Keep any supervised reduction inside the training boundary, and record the failure as a note about this artifact, not a rule against reduction.

Where the conclusion applies

Constructed embeddings with a label that depends on exactly two directions, a logistic regression, and PCA fitted on 600 rows. The chapter's failed reduction used a real pretrained artifact that is not reproduced here. In this generator a reduction that keeps the label also beats the unreduced model, because it drops 52 directions that only add estimation noise; real embeddings need not behave that way.

Check your understanding: Why does PCA with 12 components beat the unreduced model here, and what does that say about banning PCA?
Twelve components include the two label directions and drop most of the 52 directions that carry no label, which an unreduced fit has to estimate from 600 rows (0.806 against 0.784). The same reduction scores chance at 8 components, so the verdict belongs to this artifact and this k, as the chapter says: it is a comparison to run, not a ban.

Chapter 11 source: section "Failure 1: PCA on Pretrained Features".