Part I · Chapters 1 to 5
The Foundation
- 011 demonstration
What Actually Wins on Kaggle
When the model gets more capacity on the same columns, which assessment rewards the upgrade, and which one tracks the score on subjects the model never saw?
- 021 demonstration
The Process Mindset
How much of the winning version's public gain is still there on rows the choice never used, as the number of logged versions grows?
- 031 demonstration
Building a CV That Matches the Test Set
When the relationship between features and outcomes changes from season to season, how far does each validation scheme's Brier score sit from the score on the next season?
- 041 demonstration
Tracking the CV-LB Gap
With no shift and no leak, how far does the CV-LB gap move from sampling alone, and can a public sample of a given size reveal a candidate that is truly better by a small amount?
- 051 demonstration
Adversarial Validation
When train and test differ, does the domain AUC tell you whether the target model's score has changed?
Part II · Chapters 6 to 11
Feature Engineering and Domain Knowledge
- 061 demonstration
Running Case: March Mania 2026
Does an Elo decay chosen on past seasons find how much team strength truly carries over, and does it beat the chapter's 0.65?
- 071 demonstration
Domain Features as Testable Hypotheses
How much does an entity-history feature add over the raw amount when the key merges several customers, and how much of its apparent value comes from rows the prediction could not see?
- 081 demonstration
The Five-Stage Feature Engineering Process
When the target is multiplicative and skewed, does fitting log1p(y) help, and does the answer depend on the metric and on how the prediction is brought back to the raw scale?
- 091 demonstration
Target Encoding Without Leakage
How far does each target-encoding pipeline's cross-validation estimate drift from its score on new rows?
- 101 demonstration
Interactions, Groups, and Text
As more candidate pairs are screened, does the search score keep describing what the selected features earn on new rows?
- 111 demonstration
When Feature Engineering Fails
How many principal components does a reduction need before it keeps the label, and what does the variance kept say about that?
Part III · Chapters 12 to 15
Models and Tuning
- 121 demonstration
The GBM Trio
How much does it matter whether a category is fed as ordered integer codes, as native categorical splits or as one-hot columns?
- 131 demonstration
Hyperparameter Tuning with Optuna
As the trial budget grows, how much of the search's reported improvement shows up on rows the search never used?
- 141 demonstration
What Doesn't Work
When the hidden grouping matters to the label, how much of that value does a K-means column recover for a boosted-tree model?
- 151 demonstration
When the Problem Isn't Tabular
How good must a reviewer be before reviewing every answer beats the base model, and does routing change that?
Part IV · Chapters 16 to 18
Ensembling and Selection
- 161 demonstration
The Fourth-Root Blend
How alike can three models' predictions be before a blend stops beating the best single model, and do the weights matter?
- 171 demonstration
Stacking and Hill-Climbing
As the candidate library fills with near-duplicate models, how much of the hill-climbing improvement shows up on assessment rows?
- 181 demonstration
Final Submission Selection
How much public evidence does it take before the public leaderboard picks as well as a local validation set?
Part V · Chapters 19 to 21
Case Studies in Depth
- 191 demonstration
March Mania: Prediction at Scale
If the team history quietly includes later seasons, would the development score tell you, and would the label-flip test?
- 201 demonstration
Horse Health: Small Data, Big Discipline
How much of the accuracy a tuned class factor adds on its own OOF rows survives on new rows?
- 211 demonstration
Financial Competitions: Time, Regime, and Leakage
How far does each validation scheme's estimate drift from the model's score on the future as the label horizon grows?
Part VI · Chapters 22 to 41
Advanced Techniques
- 221 demonstration
Vision and Multimodal: When You Need a VLM
If the label signal sits low in the embedding's variance ranking, what does a PCA that keeps 90% of the variance leave for the second stage?
- 231 demonstration
Sensors and HAR: The Cross-Subject Problem
How far above the score on new people does a random stratified split sit, and does leave-one-subject-out close the gap?
- 241 demonstration
Building a Kaggle Portfolio
How much of a blend's gain is lost, and what does it look like, when a teammate's OOF file is joined by position but is partly out of order?
- 251 demonstration
The Continuous Learning System
How much of the noise behind a logged gain does averaging more fold seeds remove, and what does a seed-based threshold then keep?
- 261 demonstration
Advanced Target Encoding
How much smoothing does a target encoding need, and does the answer change with how strongly categories really differ?
- 271 demonstration
SHAP as a Feature Engineering Signal
When two columns carry the same information, does the one with low attribution deserve to be dropped, and what does dropping both cost?
- 281 demonstration
Nearest-Neighbor Features
How many neighbours does it take before including the row itself stops distorting the estimate, and what does the legitimate feature gain?
- 291 demonstration
Advanced Feature Selection
Which columns does each screen keep, and at what null percentile does a strict screen start discarding real signal?
- 301 demonstration
External Data and Leakage
How far does a validation score built on revised data drift from deployment, and when does the training join start to cost accuracy too?
- 311 demonstration
Deep Learning for Tabular Data
Does a neural model that is worse on its own lower the blend's error, and does that survive as the tree model gets more data?
- 321 demonstration
Denoising Autoencoders for Tabular Pretraining
Does a lower reconstruction error mean the pretrained features help a small labeled model more than an untrained encoder would?
- 331 demonstration
Data Augmentation for Tabular Data
How much does resampling before the split inflate cross-validation, and does resampling inside the fold beat weighting or doing nothing?
- 341 demonstration
Pseudo-Labeling
How much does the shortcut teacher inflate the student's cross-validation score, and what does honest pseudo-labeling add?
- 351 demonstration
Memory Optimization for Large Datasets
At what ID size does a blanket float32 downcast start merging customers, and how much memory does it save compared with a checked downcast?
- 361 demonstration
Categorical Embeddings
How large can the share of unseen categories get before an integer label code is worse than dropping the column, and what does the unknown rule cost a target encoding?
- 371 demonstration
Stacking Deep
How far does a meta-model's score on its own OOF rows drift from an honest assessment as the library of base models grows, and when does a regularized meta-model overtake the unregularized one?
- 381 demonstration
Calibration and Post-Processing
How large must the development set be before isotonic calibration beats the simpler Platt calibrator on new rows?
- 391 demonstration
Imbalanced Data
How much log loss does negative downsampling cost, and at what keep rate does the analytic correction stop being as good as calibrating on development rows?
- 401 demonstration
Knowledge Distillation
Do soft teacher targets beat hard labels for a small neural student, or does that depend on how well the hard-label baseline is tuned?
- 411 demonstration
Online Learning
How much of an adaptive model's advantage over a frozen model survives as labels arrive later, and how wrong is a replay that ignores the delay?
Part VII · Chapters 42 to 65
Specialized Domains and the Landscape
- 421 demonstration
Image Augmentation
When does adding transformed copies of the training images raise held-out accuracy, and when does it lower it?
- 431 demonstration
Image Classification Tricks
How much does a random split of crops overstate accuracy compared with splitting by source image, and how does that grow with crops per image?
- 441 demonstration
Loss Functions for CV
Does focal loss beat cross-entropy on the metric that is scored, and what does it cost elsewhere?
- 451 demonstration
Metric Learning and ArcFace
How much does the angular margin improve retrieval of unseen identities over a cross-entropy descriptor, and where does a larger margin stop helping?
- 461 demonstration
Segmentation Architectures
Which threshold and post-processing rule is best depends on how Dice is averaged: how large is that effect, and when does it appear?
- 471 demonstration
Medical Imaging Patterns
How far does image-level cross-validation overstate the score on new patients, and does grouping by patient remove the error?
- 481 demonstration
Pseudo-Labeling for CV
Does a stricter confidence threshold make pseudo-labeling pay, and does it remove the leak from a teacher that saw the validation labels?
- 491 demonstration
Text Feature Engineering
At what typo rate, and at what term coverage, does a character TF-IDF model overtake a word model?
- 501 demonstration
LLM Fine-Tuning for Kaggle
How does the rank of a LoRA-style update change held-out error against the original checkpoint and a full update when target data is small?
- 511 demonstration
Time Series Cross-Validation
How much does a backtest that ignores label availability flatter each recipe, and does it still rank the recipes correctly?
- 521 demonstration
Time Series Feature Engineering
How much does a backtest flatter a model whose lag features read days that would not yet exist at the forecast origin?
- 531 demonstration
Financial Competition Patterns
When does removing factor exposure from the features help the score, and when does it cost correlation?
- 541 demonstration
Combinatorial Optimization
With a fixed number of evaluated moves, does a hotter starting temperature or a better starting tour give the shorter final tour?
- 551 demonstration
Pancake Sorting: A Constructed Search Example
Under the same beam width, how many instances does each ranking actually solve, and how far from optimal are its solutions?
- 561 demonstration
Reinforcement Learning for Game Competitions
At what size of intermediate reward does a learner stop trying to win, and does the shaped return reveal it?
- 571 demonstration
PatchTST for Time Series and HAR
Does patching beat a feature baseline at the same forward origins, and how does the patch length change the answer and the token count?
- 581 demonstration
Sensor Embeddings and Augmentation
When does a left-right swap help a wearable activity classifier, and what does it cost when the label is not updated with the swap?
- 591 demonstration
Threshold Optimization
How far does the development F1 of a tuned threshold overstate its F1 on fresh labels, and how does that depend on the size of the development set?
- 601 demonstration
Multi-Target Learning
When does a dense auxiliary target improve a scarce main target on new rows, and how much does the loss weight matter?
- 611 demonstration
Playground Series Strategy
How much of the public leader's visible advantage is still there on private rows as more near-equal candidates are screened?
- 621 demonstration
The Kaggle Landscape, 2024 to 2026
At what deadline does a stronger model stop being the better pipeline, and what does a retained feasible reference recover?
- 631 demonstration
Lessons from the Expanded Winning-Solution Corpus
When winners use a technique more often than the field does, how much does that say about the technique's effect?
- 641 demonstration
Audio and Signal Competitions
How far does chunk-level cross-validation overstate accuracy on new recordings, and does it distort the comparison between models?
- 651 demonstration
Recommendation Systems
As the candidate cap grows, how much of the rising oracle score does a trained reranker actually capture?
More for this book
A worksheet for every chapter: predict first, then inspect the worked notebook and the lessons that apply.
Search all 561 reviewed conditional lessons by application, workflow, modality and reader.
Three paths through the chapters, each ending in a concrete artifact.
Read the sourced corrections before adapting the book examples.
Notebooks and skills on GitHub
Every chapter has a Jupyter notebook that reproduces its calculations, and a skill an AI assistant can use to apply the chapter's method to your own project.