Winning Kaggle the Reproducible Way

Chapter 44

Loss Functions for CV

Name the failure first, keep cross-entropy as the comparator, and judge a specialised loss on the metric that is scored.

A focal loss changes what the model is trained to fix. One demonstration trains the same small network with cross-entropy and with focal loss on a rare class, and measures the scored ranking metric, the F1 at a default cutoff and the quality of the probabilities.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Focal loss against its cross-entropy comparator on a rare class, by gamma

Does focal loss beat cross-entropy on the metric that is scored, and what does it cost elsewhere?

The factor (1 - p_t)^gamma shrinks the loss of rows the model already classifies well, so training spends its effort on hard rows, including the hidden subgroup of positives. The same factor also shrinks easy positives, and it stops the probabilities from matching frequencies: they drift upward on a rare class. Raise gamma and watch the ranking gain peak and turn into a loss while the log loss only gets worse.

Equation: Focal loss equals minus one minus p t, raised to the power gamma, times the natural log of p t

Scroll sideways for the whole equation

p_t is the probability the model gives the true class, gamma the focusing parameter (0 gives cross-entropy), FL the focal loss for one row and AP the average precision of the ranking.

Predict first. Compared with cross-entropy, what does focal loss with gamma = 2 do to the log loss on fresh rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Focal loss against its cross-entropy comparator on a rare class, by gamma. Two panels. Left, paired bars of average precision, tuned-cutoff F1 and default-cutoff F1 for cross-entropy and for focal loss with gamma 2. Right, mean predicted probability for each loss against the true rate of 0.050; the focal model's mean is 0.191.
Focusing parameter gamma (0 is ordinary cross-entropy): 2: the common starting value
Constructed example: seeded synthetic rare-class data and a numpy network, measured by the chapter activity.

Calculated values

Average precision, cross-entropy
0.301
Average precision, focal
0.322
Paired gain (standard error)
+0.021 (0.003)
F1 at cutoff 0.5, focal
0.198
Log loss, cross-entropy
0.155
Log loss, focal
0.256

At gamma 2, average precision is 0.322 against 0.301 for cross-entropy: 0.322 - 0.301 = 0.021 (standard error 0.003, better in 94% of the draws). The ranking improves. At the default cutoff F1 moves from 0.218 to 0.198; with a cutoff tuned on development rows it moves from 0.341 to 0.350. Log loss goes from 0.155 to 0.256 because the mean predicted probability is 0.191 on a 0.050 positive rate.

Worked steps

  1. Average precision: 0.322 - 0.301 = +0.021 (focal minus cross-entropy).
  2. F1 at the default cutoff: 0.198 - 0.218 = -0.020.
  3. F1 at the tuned cutoff: 0.350 - 0.341 = +0.009.
  4. Log loss: 0.256 - 0.155 = +0.101.

Use the idea

Keep an unweighted cross-entropy run as the reference, compare focal values on the scored metric over several seeds, and check the probabilities of a focal model on development rows before trusting them: if the score needs probabilities, fit a calibrator on separate development rows.

Where the conclusion applies

Constructed data, a small numpy network standing in for a CNN and a binary focal loss without the optional alpha weights. The ranking gain at gamma = 2 is a property of this generator. A side check without the hard subgroup still showed a smaller gain, so the subgroup explains only part of it; the rest is likely focal loss damping overconfident fits on about 100 positives. Both losses share one optimiser setting; neither was retuned. On other data the gain can be zero or negative, which is why the chapter asks for the comparison rather than prescribing gamma. Cutoffs are tuned for F1 only.

Check your understanding: At gamma = 5 the F1 at the default cutoff collapses while the tuned-cutoff F1 barely moves. What does that say about comparing losses?
Strong focusing squeezes the probabilities into a narrow middle band (mean 0.332), so almost nothing reaches 0.5 (0.003 of rows are flagged), so F1 at the default cutoff falls from 0.218 to 0.080, yet with a cutoff tuned on development rows F1 is 0.340 against 0.341. A comparison is only fair when each loss gets its own decision rule: tuning the cutoff alone lifts the cross-entropy F1 by 0.123, while gamma = 2 adds only about 0.01 on top of a tuned cutoff and 0.021 average precision.

Chapter 44 source: section "Focal Loss for Imbalanced Classes".