Winning Kaggle the Reproducible Way

Chapter 45

Metric Learning and ArcFace

Separate the training head, the descriptor and the ranking rule, and judge the margin on identities the model never saw.

A classifier over training identities cannot name a new one, but its descriptor can still retrieve it. One demonstration trains an ArcFace-style head and a plain cross-entropy head and measures retrieval of identities that were never in training, by angular margin.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

ArcFace margin against a cross-entropy descriptor, scored on unseen identities

How much does the angular margin improve retrieval of unseen identities over a cross-entropy descriptor, and where does a larger margin stop helping?

The margin makes the training target harder, so the embedding has to put each identity's images at a smaller angle to its prototype than a plain cross-entropy descriptor does. That compactness carries over to identities that were never trained on, because retrieval compares angles and never uses the head. Normalising with no margin (m = 0) already gives part of the gain over the plain head; the margin adds the rest. A margin that is too large asks for more room between 40 prototypes in 8 dimensions than exists, and training becomes unreliable.

Equation: The target logit z y equals s times the cosine of theta y plus m; every other logit z c equals s times the cosine of theta c

Scroll sideways for the whole equation

theta_y is the angle between an embedding and the prototype of its true identity, m the additive angular margin in radians and s the feature scale. MAP@5 averages, over query images, the reciprocal rank of the true identity among the five best distinct gallery identities (0 if absent).

Predict first. Which margin gives the best retrieval of unseen identities, and what happens at the largest margin of 1.2?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: ArcFace margin against a cross-entropy descriptor, scored on unseen identities. Two panels. Left, a paired dot plot of unseen-identity MAP@5 for each of 16 draws, cross-entropy against ArcFace with margin 0.5, with diamonds at the means 0.640 and 0.778. Right, MAP@5 for the two heads on seen and on unseen identities.
Angular margin m (radians): 0.5: the chapter's starting value
Constructed example: seeded synthetic identity features and a numpy ArcFace head, measured by the chapter activity.

Calculated values

Reference MAP@5, unseen
0.640
ArcFace MAP@5, unseen
0.778
Paired gain (standard error)
+0.139 (0.012)
ArcFace wins in draws
100%
ArcFace worst draw
0.703
ArcFace MAP@5, seen
0.771

With margin 0.5, retrieval of unseen identities reaches MAP@5 0.778 against 0.640 for the cross-entropy descriptor: 0.778 - 0.640 = 0.138 (standard error 0.012, better in 100% of the 16 draws). The margin head is clearly better than the reference. On identities seen in training the margin head scores 0.771, close to its unseen score. The reference scores 0.634 on seen identities.

Worked steps

  1. Unseen-identity MAP@5, margin head minus reference: 0.778 - 0.640 = +0.138.
  2. Seen-identity MAP@5, margin head minus reference: 0.771 - 0.634 = +0.137.
  3. Paired standard error of the unseen gain: 0.012; the margin head wins in 100% of draws.
  4. Worst single draw of the margin head: 0.703 (reference worst: 0.569).

Use the idea

Build a reference descriptor with plain cross-entropy, add the margin head as a separate comparison, score both with query identities held out of training, and treat the margin as a setting to tune, not a constant to copy.

Where the conclusion applies

Constructed features rather than images, a linear embedding standing in for a backbone, and s = 16 instead of the chapter's example 64 because there are only 40 classes. The cross-entropy reference uses an unnormalised linear head and is scored on the same normalised embedding. Seen and unseen identities score almost the same here, because a linear embedding does not memorise identities; a deep backbone can, so keep reporting them separately.

Check your understanding: Margins 0.5 and 0.8 retrieve almost equally well. Which would you carry into a real competition, and why not simply the largest margin?
Take the smaller of the two, because the gain is flat between them (0.778 against 0.781) while the failure region begins somewhere above: at m = 1.2 the average gain falls to 0.047 with a standard error of 0.032, and the worst draw scores 0.452. A margin is tuned on development identities and then frozen, and the chapter lists larger margins as a source of unstable optimisation.

Chapter 45 source: section "Two Head Settings: Scale and Margin".