Winning Kaggle the Reproducible Way

Chapter 47

Medical Imaging Patterns

Keep every image from a patient in one fold, and check that no patient crosses the boundary.

Images from one patient resemble each other, so an image-level split lets a model recognise patients instead of disease. One demonstration measures how far image-level and patient-grouped cross-validation drift from the score on fresh patients as the number of images per patient grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Image-level and patient-grouped cross-validation against fresh patients, by images per patient

How far does image-level cross-validation overstate the score on new patients, and does grouping by patient remove the error?

When a patient's images are split across folds, the model sees the patient's anatomy in training and meets it again in validation, so it can read the label from who the patient is. With one image per patient nothing can be shared and the two schemes agree. Each added image per patient puts more near-copies across the fold boundary and the image-level estimate climbs, while the grouped estimate stays near the fresh-patient score.

Equation: error equals the cross-validation AUC minus the AUC on fresh patients

Scroll sideways for the whole equation

AUC is the area under the ROC curve of the pooled out-of-fold probabilities. The error of a validation scheme is its AUC minus the AUC of the same model on fresh patients.

Predict first. With 4 images per patient, how does image-level 5-fold AUC compare with the AUC on fresh patients?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Image-level and patient-grouped cross-validation against fresh patients, by images per patient. Two panels at 4 images per patient. Left, bars of AUC for image-level folds (0.874), patient-grouped folds (0.674) and fresh patients (0.707), with a dot for each of 8 cohorts. Right, the share of validation images whose patient is also in training: 98.8% image-level and 0.0% grouped.
Images per patient: 4
Constructed example: seeded synthetic patient cohorts and an extremely randomized trees classifier, measured by the chapter activity.

Calculated values

Image-level 5-fold AUC
0.874
Grouped AUC
0.674
AUC on fresh patients
0.707
Image-level error
+0.167
Grouped error
-0.033
Patients shared across folds (image-level)
98.8%

With 4 images per patient the same model scores AUC 0.707 on fresh patients. Image-level folds report 0.874: 0.874 - 0.707 = 0.167 (their error). Grouped folds report 0.674, which is -0.033 from the fresh score. 98.8% of image-level validation images belong to a patient that is also in training, against 0.0% when grouped. The image-level number measures how well the model recognises patients it has already seen.

Worked steps

  1. Image-level: 0.874 - 0.707 = +0.167 (estimate minus fresh patients).
  2. Grouped by patient: 0.674 - 0.707 = -0.033.
  3. Image-level minus grouped: 0.874 - 0.674 = +0.200.
  4. Patient overlap between training and validation: 98.8% (image-level), 0.0% (grouped).

Use the idea

Split by patient (or slide, or study) before anything else, assert that no patient appears in both a training and a validation fold, and read an image-level estimate on multi-image data as a measure of memorised patients.

Where the conclusion applies

Constructed features and a patient-level label, so every image of a patient shares the answer; with a label that varies within a patient (a lesion on some slices only) the leak is smaller. The tree model memorises easily; a regularised model leaks less. Grouped folds run slightly below the fresh-patient score here, by up to about 0.04 at eight images. A side check found that training on 120 rather than 150 patients does not explain it; with only 150 independent patients per cohort the grouped estimate is noisy, and the small gap changes sign under other seeds. It is a property of this setup, not a rule.

Check your understanding: With one image per patient the two splits agree, and with eight they differ by about 0.26 AUC. What should you check before trusting any image-level estimate?
How many images each patient contributes and whether any patient crosses the boundary. At one image per patient the image-level AUC is 0.700 against 0.701 grouped, but at eight it is 0.924 against 0.667 grouped while fresh patients score 0.706: the error grows from nothing to 0.218 purely from repeated patients.

Chapter 47 source: section "Keep all images from each patient in one validation fold".