The Grantham projectCurriculum learning, tested 200 times
Train the same model on the same examples in four different orders and see what order really does.
The learner downloads four census tables from Nomis for all 466 output areas in South Kesteven and builds six numbers for each: population density on a log scale, and the shares of one-person households, residents aged 65 or over, detached homes, flats and children under 15. The label says whether the area lies inside one of the five towns in the table above: 321 do and 145 do not. A model that always answered "town" would be right 68.9% of the time, so that is the score to beat.
The model is logistic regression, trained one example at a time by gradient descent, which makes the order of examples matter. To rank examples from easy to hard the learner first trains a "teacher" model on the training data and records how confident it is about each example's true label. Then four students are trained on the same 326 training areas in different orders: random, easiest first, easiest first but alternating town and country, and hardest first. Each student is scored on 140 areas it has never seen, after 10, 25, 50, 100, 200 and all 326 examples, then again after four more passes in random order. The whole thing is repeated on 200 random splits of the data.
Accuracy on unseen areas after a given number of training examples, mean of 200 splits (our Python)| Order of examples | After 10 | After 50 | After 100 | After 326 | After five full passes |
|---|
| Random | 79.4% | 86.8% | 88.1% | 88.2% | 88.1% |
| Easiest first | 68.5% | 74.8% | 77.1% | 86.9% | 88.2% |
| Easiest first, town and country alternated | 71.2% | 73.1% | 75.2% | 83.0% | 88.1% |
| Hardest first | 21.3% | 20.2% | 72.7% | 88.3% | 88.2% |
Easy first did not help. It was behind random order after 50 examples in every one of the 200 splits. Part of the reason is visible straight away: the 40 easiest areas, ranked by a teacher trained on all 466, are 82.5% town, so an easy-first model spends its early lessons on one class. Alternating town and country fixes that and still loses to random, in 199 of 200 splits at 100 examples. Our reading is that the easy examples sit far from the boundary between town and country: they show which way the line should face but not where it should go, and the examples that pin it down come last.
Hardest first was dramatic in the other direction. After 10 to 50 examples it scored between 16% and 21%, far worse than always guessing "town", in all 200 splits. The hardest examples are the ones that break the pattern, town areas that look rural and village areas that look urban, so a model that learns from them first learns the rule backwards. And the ending matters as much as the start: after five full passes every order finished between 88.1% and 88.2%, and easy first beat random in 73 splits while losing in 74. Order changed how the model got there, not where it ended up.
Ages 8 to 11
Teach a friend to sort town and country pictures, once starting with obvious ones and once with odd ones, and compare.
Ages 11 to 15
Train a tiny model in Python on the census areas and plot its accuracy as each example arrives.
Ages 15 and up
Build the teacher, the four orders and the 200 splits, then design a curriculum that actually beats random.
Data and honesty notes
Census 2021 tables TS006, TS017, TS007A and TS044 via Nomis, and the ONS output area to built-up area lookup, under the Open Government Licence. The label, the features, the models and every percentage are ours. A different model, a smaller learning rate or a different idea of "easy" could give a different result; the page reports what this setup did, not a law of machine learning.
