The Bootle projectLearning curves on 942 Sefton census areas
Two models, seven training sizes, thirty repeats of each, and one chart that says when to stop collecting.
From the Nomis API the learner downloads four Census 2021 tables for every output area in Sefton: car or van availability, household size, population density and accommodation type. The borough has 942 output areas and 123,018 households, of which 32,443 had no car or van. The task is to predict, for an area the model has not seen, the share of households without a car, using four clues: density, the share of one-person households, the share of detached homes and the share of purpose-built flats. Two hundred areas are set aside as a test. Training sets of growing size are drawn from the other 742, and the whole experiment is repeated thirty times with fresh random draws.
Mean absolute error in percentage points, average of 30 repeats, our Python run on Census 2021 data from Nomis| Areas used for training | Linear model, unseen areas | Linear model, training areas | Forest, unseen areas | Forest, training areas |
|---|
| 10 | 13.66 | 5.22 | 9.83 | 3.82 |
| 20 | 10.16 | 6.86 | 8.85 | 3.36 |
| 50 | 8.94 | 8.07 | 7.93 | 3.02 |
| 100 | 8.51 | 8.17 | 7.56 | 2.79 |
| 200 | 8.38 | 8.12 | 7.18 | 2.59 |
| 400 | 8.32 | 8.10 | 6.89 | 2.50 |
| 700 | 8.31 | 8.18 | 6.77 | 2.44 |
A model that ignores the clues and always answers with the average of its training areas is wrong by 13.88 points, so that is the bar to beat. With ten areas the straight-line model barely beats it, at 13.66, while looking excellent on the ten areas it memorised. By 100 areas its two curves have nearly met, and from 200 to 700 the error on unseen areas moves only from 8.38 to 8.31. That flat line is a plateau: the linear model has learned all that a straight-line rule can express, and tripling the data buys almost nothing. The random forest of 200 trees tells a different story. It is still improving at 700 areas, 6.77 against 6.89 at 400, and the wide gap between its training error of 2.44 and its test error shows a flexible model that would use more data if Sefton had more areas to give.
The repeats matter as much as the averages. At ten training areas the linear model's test error swung widely from one random draw to another, with a standard deviation above 5 points; at 700 it was 0.4. A single run on a small sample can flatter or insult a model by accident, which is why each point on a learning curve should be an average.
Ages 8 to 11
Guess a rule from three examples, then from ten, and count the misses on cards you have not yet turned over.
Ages 11 to 15
Fit a line in Python to 20 Sefton areas and to 200, then chart both errors side by side.
Ages 15 and up
Code the full experiment: held-out test set, repeats, two models and the finished learning curve.
What is published and what is ours
The household counts are Office for National Statistics Census 2021 data, fetched from Nomis under the Open Government Licence. The models, the error figures and the curves are our own work and appear in no official release. The project predicts a share for an area and makes no statement about any household or about why areas differ.
