Winning Kaggle the Reproducible Way

Chapter 46

Segmentation Architectures

Fix the Dice reduction, the empty-mask rule and the threshold before comparing models, and keep each choice on development images.

Pooled Dice and mean per-image Dice can rank the same masks very differently. One demonstration tunes a threshold for each, adds a minimum-size rule, and scores all three on held-out images as the share of empty images grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Three mask rules scored by pooled and per-image Dice, by the share of empty images

Which threshold and post-processing rule is best depends on how Dice is averaged: how large is that effect, and when does it appear?

Pooled Dice is dominated by the pixels of large objects, so the best threshold under it tolerates speckle in empty images. Per-image Dice gives every empty image a score of 0 or 1, so the same speckle costs a whole image each time. The two reductions pick different thresholds, and the per-image threshold climbs as the share of empty images grows. A minimum-size rule removes small speckle without raising the threshold that real objects need.

Equation: Pooled Dice sums intersections and supports over all images before dividing; per-image Dice divides within each image and averages the N ratios

Scroll sideways for the whole equation

P and Y are the predicted and true masks of one image, and Dice = 2|P and Y| / (|P| + |Y|). Pooled Dice sums |P and Y|, |P| and |Y| over all images before dividing; mean per-image Dice divides within each image and then averages, scoring 1 when both masks are empty.

Predict first. With half the images empty, what mean per-image Dice does the pooled-Dice threshold reach on held-out images?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Three mask rules scored by pooled and per-image Dice, by the share of empty images. Two panels at 50% empty images. Left, held-out per-image and pooled Dice for three rules. Right, both Dice values against the probability threshold, with dashed lines at the thresholds the pooled and per-image rules chose, 0.630 and 0.825.
Share of images with an empty mask: 0.5: half are empty
Constructed example: seeded synthetic masks and a generated probability map, measured by the chapter activity.

Calculated values

Pooled rule, per-image Dice
0.723
Pooled rule, pooled Dice
0.894
Per-image rule, per-image Dice
0.801
Minimum-size rule, per-image Dice
0.872
Minimum-size rule, pooled Dice
0.903
Gain over pooled rule (SE)
+0.149 (0.009)

With 50% of images empty, the threshold tuned for pooled Dice (0.630) scores pooled Dice 0.894 but mean per-image Dice 0.723. Tuning for per-image Dice moves the threshold to 0.825 and gives 0.801 per image and 0.855 pooled. Adding a minimum object size (mean 7.0 pixels) reaches 0.872 per image: 0.872 - 0.723 = 0.149 over the pooled-Dice rule (standard error 0.009, better in 100% of draws).

Worked steps

  1. Pooled-Dice rule, per-image minus pooled: 0.723 - 0.894 = -0.171.
  2. Per-image rule, pooled Dice: 0.855 - 0.894 = -0.039 against the pooled rule.
  3. Minimum-size rule over the pooled-Dice rule, per image: 0.872 - 0.723 = +0.149.
  4. Thresholds chosen: pooled 0.630, per-image 0.825, with minimum size 0.590.

Use the idea

Write down the official reduction and empty-mask rule first, choose the threshold and any minimum-size rule on development images with that exact metric, and report the pooled number only as a second view.

Where the conclusion applies

Constructed images and a generated probability map stand in for a network. The speckle in this generator is made of small blobs, which is why a minimum-size rule works so well here; real false positives can be large and need a different remedy. One disc per image, a single class, and a threshold grid of 15 values.

Check your understanding: Tuning the threshold directly for per-image Dice fixes that, yet pooled Dice falls. What repairs both, and what does it depend on?
The per-image threshold rises to 0.825 to suppress speckle, which also shrinks real masks, so pooled Dice falls from 0.894 to 0.855. A minimum-size rule keeps the threshold near 0.59 and deletes components under about 7 pixels, reaching 0.872 per-image and 0.903 pooled. That works here because the speckle blobs are small; it is a property of this generator.

Chapter 46 source: section "Segmentation Losses".