The Lowestoft projectDoes data augmentation really help? Two histories as a test
Teach a model to tell Gillingwater from Longe with very few examples, then add altered copies and measure fairly.
The learner splits each book into sentences, keeping those of 8 to 60 words and leaving out later additions and footnotes: 1,922 sentences from Gillingwater and 938 from Longe. A simple word-count classifier, known as naive Bayes, learns which words each author favours. To imitate a shortage of data, it is trained on only a handful of sentences per author and tested on 300 unseen sentences from each. The augmented version gets four extra copies of every training sentence, each with about one word in ten deleted and a pair of words swapped. Longe's remark that the oldest record naming the town is Domesday Book, lowercased as "the most ancient record in which we find any mention of lowestoft is domesday book" becomes, for example, "the most ancient record which of find any mention we lowestoft is domesday book". Every result is averaged over 200 random splits.
Mean accuracy on 600 unseen sentences, 200 random splits, our Python run, 28 September 2026| Training sentences per author | Without augmentation | With augmentation |
|---|
| 5 | 55.1% | 55.7% |
| 10 | 57.6% | 58.4% |
| 20 | 61.8% | 62.6% |
| 50 | 68.6% | 68.9% |
At first glance augmentation works. It adds a little under a point at every size, and at 20 sentences per author it wins in 133 of the 200 splits and loses in 56. A beginner might stop there and report success. The careful next step is a control: what happens if, instead of altered copies, the program simply adds four exact copies of each sentence? That changes nothing about the information available, only how heavily the training examples count.
Controls at 20 training sentences per author, same 600 test sentences and 200 splits| Training data | Mean accuracy |
|---|
| Original sentences only | 61.8% |
| Plus four exact copies of each | 62.6% |
| Plus four copies with words swapped only | 62.6% |
| Plus four copies with words deleted only | 62.6% |
| Plus four copies with both changes | 62.6% |
Every version lands on the same figure. Exact copies do just as well as the clever ones, so the small gain comes from giving the training examples more weight, not from new information. The swapped copies are the clearest case: their results match the exact copies in every one of the 200 splits, because a word-count model never sees word order at all. An augmentation only helps if it changes something the model can actually notice.
The leakage trap, 95 original sentences per author, 80% train and 20% test| Order of steps | Mean accuracy |
|---|
| Augment all sentences, then split into train and test | 99.2% |
| Split first, then augment only the training sentences | 71.5% |
The second trap is far more dangerous. If every sentence is augmented first and the copies are then shuffled into training and test sets, near-identical versions of test sentences sit in the training data. The model recognises them and scores 99.2%, a result that looks spectacular and means nothing. Split first, augment only the training part, and the honest figure is 71.5%. Order matters: set the test data aside before doing anything else.
Ages 8 to 11
Run a fair-test experiment with a control group and see why the control matters.
Ages 11 to 15
Split two texts into sentences in Python and count which words each writer prefers.
Ages 15 and up
Build the augmentation, add the copy control, then reproduce and fix the leakage.
Their histories, our experiment
The texts are the Project Gutenberg transcriptions of Gillingwater's History of Lowestoft and Lowestoft in Olden Times by Francis Davy Longe. The sentence splits, the classifier, the augmentations and every accuracy figure are our own work.
