Winning Kaggle the Reproducible Way

Chapter 49

Text Feature Engineering

Check term overlap first, keep a sparse baseline, and compare word, character and combined representations on the same rows.

A vocabulary fitted on clean training text meets typos in the assessment text. One demonstration measures the frequency-weighted term coverage and compares word, character and combined TF-IDF models as the typo rate rises.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Word, character and combined TF-IDF when only the assessment text has typos

At what typo rate, and at what term coverage, does a character TF-IDF model overtake a word model?

A word model can only use words it saw in training, so every misspelled word becomes a blank. Character n-grams keep matching the unchanged fragments, but spanning word boundaries they also add noisy features, which is why they lose on clean text. Where the two lines cross is a property of the data and the metric, not a coverage constant. Placing both representations side by side gives the lowest log loss up to moderate noise here, but under heavy typos it only ties the character model, and with any typos it is less accurate than characters alone, so the scored metric decides.

Equation: coverage equals the count of assessment tokens found in the training vocabulary divided by the count of assessment tokens

Scroll sideways for the whole equation

Coverage is the frequency-weighted share of assessment tokens found in the training vocabulary. Log loss is the mean negative log probability of the true class (lower is better), accuracy the share of correct top predictions.

Predict first. With 20% of assessment words misspelled (about 81% coverage), which model has the lower log loss?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Word, character and combined TF-IDF when only the assessment text has typos. Two panels at a 20% typo rate. Left, log loss of word, character and combined TF-IDF models (0.434, 0.425, 0.389) with a dot for each of 8 corpora. Right, the accuracy of the same three models.
Share of assessment words that are misspelled: 0.2
Constructed example: seeded synthetic titles from invented words and sparse logistic regressions, measured by the chapter activity.

Calculated values

Token coverage
0.814
Word log loss
0.434
Character log loss
0.425
Combined log loss
0.389
Word accuracy
0.864
Character accuracy
0.890

With 20% of assessment words misspelled, 0.814 of assessment tokens are in the training vocabulary. Word log loss is 0.434 and character log loss 0.425: 0.434 - 0.425 = 0.009 (standard error 0.004). The character model is ahead on log loss by a small margin, between two and three standard errors: this is near the switch. Accuracy is 0.864 for words and 0.890 for characters; the combined model has log loss 0.389 and accuracy 0.879.

Worked steps

  1. Coverage of assessment tokens: 0.814.
  2. Log loss, word minus character: 0.434 - 0.425 = +0.009 (positive means characters are better).
  3. Accuracy, character minus word: 0.890 - 0.864 = +0.026.
  4. Combined model against the better single model: 0.389 - 0.425 = -0.037 log loss.

Use the idea

Measure frequency-weighted coverage of the assessment text first, then compare word, character and combined sparse models on identical rows with the scored metric, before spending time on embeddings.

Where the conclusion applies

Constructed text from invented words, with typos in the assessment text only; real noise has structure (keyboard neighbours, language), and training text with the same typos would favour the word model. The character analyzer spans word boundaries as in the chapter's baseline; a word-bounded analyzer would lose less on clean text. Models are regularised logistic regressions with C = 10.

Check your understanding: On clean text the word model wins clearly. Why is that not a reason to drop character features, and what does a coverage rule miss?
Which model wins changes with the noise: the word model leads by 0.051 log loss on clean text, but at 40% typos the character model leads by 0.083 and is right 0.887 of the time against 0.811. At 10% typos the models already disagree by metric, with the word model ahead on log loss (0.389 against 0.413) and the character model ahead on accuracy (0.896 against 0.883). A coverage cutoff would not reproduce that: coverage is 0.905 at 10% typos and the right choice depends on the scored metric.

Chapter 49 source: section "A Small Sparse Comparator".