Winning Kaggle the Reproducible Way

Chapter 62

The Kaggle Landscape, 2024 to 2026

Quality includes completed work: budget loading and inference together, and keep a pipeline that finishes.

A stronger model that cannot finish inside the runtime cap does not improve the submission. One demonstration counts the work of loading and running a small and a large forest against a deadline and scores what each pipeline actually delivers.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Small reference, large model and reference-then-upgrade under a runtime cap

At what deadline does a stronger model stop being the better pipeline, and what does a retained feasible reference recover?

The large forest is about 0.05 more accurate on queries it finishes, but it costs about 25 times the work of the small one. Once the deadline falls below its total, queries it never reaches are answered by the majority class. At 40% that loss swamps the gain on the rest; at 80% the two roughly cancel, and which side wins depends on the size of the model's advantage (it lost here and came out slightly ahead under another seed). The reference-then-upgrade pipeline pays a small overhead to answer everything first, so no query is left unanswered, and it spends leftover work where the two models disagree most. It scored at least as well as the reference at every deadline here.

Equation: The work W equals the load work L plus the sum over n queries of c i, and must be at most the deadline D

Scroll sideways for the whole equation

W is the work a pipeline needs, L the work to load the model, c_i the work to answer query i and D the deadline in the same units. A pipeline is feasible when W is at most D.

Predict first. At a deadline of 80% of the large model's end-to-end work, the large model answers about three quarters of the queries in file order. Which pipeline scores best on all 1,000 queries?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Small reference, large model and reference-then-upgrade under a runtime cap. Left: bars of whole-test accuracy at a deadline of 80% of the large model's work: small reference 0.729, large model in file order 0.708, reference then upgrade 0.773; a dashed line marks the large model with no deadline at 0.775. Right: end-to-end work of each model in thousands of units against the deadline line.
Deadline (percent of the large model's end-to-end work): 80%
Constructed example: seeded synthetic tasks and two random forests, with work counted from the fitted trees by the chapter activity.

Calculated values

Small reference accuracy
0.729
Large model, file order
0.708
Reference then upgrade
0.773
Large model with no deadline
0.775
Queries answered by the large model
76% in file order, 71% upgraded

With a deadline of 80% of the large model's work, the large model in file order answers 76% of the queries and scores 0.708 overall, against 0.729 for the small reference: 0.729 - 0.708 = 0.021 lower. Given unlimited time it would score 0.775, 0.775 - 0.729 = 0.046 above the reference. Reference-then-upgrade scores 0.773 and upgrades 71% of queries to the large model, recovering 95% of that gain. The large model needs 25 times the small model's work.

Worked steps

  1. Large model against reference: 0.708 - 0.729 = -0.021.
  2. Gain with no deadline: 0.775 - 0.729 = 0.046.
  3. Reference-then-upgrade against reference: 0.773 - 0.729 = 0.044.
  4. Deadline in work units: 1489 thousand, against 1861 thousand needed by the large model.

Use the idea

Estimate load and per-query work on a representative slice, scale to the real test size, keep a pipeline that finishes under the cap, and upgrade queries with the leftover budget only after measuring the upgrade rule.

Where the conclusion applies

Constructed data and counted work rather than timed work: node visits stand in for seconds on one fixed machine, and the 5 units per node for loading is an assumed constant, so the deadlines are shares of one pipeline's cost and not a statement about any real GPU or CPU. Measure your own pipeline on your own hardware. Unanswered queries here get the majority class, which is kinder than a real timeout that can invalidate a submission. Ordering upgrades by the small model's uncertainty is one routing rule, and the chapter asks for each route to be assessed on its own.

Check your understanding: At 80%, reference-then-upgrade answers only 71% of the queries with the large model yet scores 0.773. How does it recover almost all of the large model's gain?
It answers every query with the small model first, which costs 4% of the large model's end-to-end work, then upgrades the queries where the small model is least sure, which is where the larger model's answers differ most. Upgrading 71% of the queries recovers 95% of the accuracy gain of the large model run on all of them. That route is a measured result for this generator, and the chapter asks that any such route be assessed with its overhead.

Chapter 62 source: section "Quality Includes Completed Work".