The Dulwich projectDouble descent: one model, grown from 5 features to 10,000
Make the model bigger step by step, and measure its error each time on areas it has never seen.
The Census divides Southwark into 959 output areas, 156 of them in the five wards above, where they hold 20,953 households. For every area in the borough we took three facts, its population density, its share of one-person households and its share of flats, and asked a model to predict a fourth: the share of households with no car. The model sees only 100 areas chosen at random and is tested on the other 859. Its size is set by one number, how many random features it builds from the three inputs. Each size is run 30 times with different random choices and the middle result is reported.
Error in percentage points by model size, 100 training areas, 859 test areas, our Python run on Census 2021 data| Features | Error on training areas | Error on unseen areas |
|---|
| 5 | 9.42 | 10.85 |
| 10 | 8.26 | 9.63 |
| 20 | 7.68 | 10.17 |
| 40 | 6.44 | 13.05 |
| 80 | 3.65 | 49.2 |
| 100 | 0.98 | 494.87 |
| 120 | 0.0 | 78.25 |
| 200 | 0.0 | 32.01 |
| 1,000 | 0.0 | 18.27 |
| 10,000 | 0.0 | 15.59 |
Read down the last column. With 10 features the model is at its most accurate, 9.63 points out on areas it has not seen. After that, more features help it on the training areas and hurt it everywhere else, which is ordinary overfitting. At 100 features, exactly one per training area, it can fit every training value almost perfectly, and to do so it has only one possible set of weights, a wild one: the typical error on unseen areas is 494.87 points, for a quantity that can only lie between 0 and 100. Past that threshold something changes. With more features than areas there are many ways to fit the training data exactly, the method picks the gentlest of them, and the error falls again: 78.25, 32.01, 18.27, 15.59.
That second fall is double descent, and it is real. But look at where it ends. At 10,000 features the error is 15.59, which is worse than the 9.63 of the ten-feature model and worse even than the 14.91 you get by ignoring the inputs and guessing the training average every time. On this small, noisy problem the giant model recovers from the peak without ever catching the small one. A tiny penalty on large weights, called ridge, removes the peak altogether, 12.55 at 100 features instead of 494.87, and still the lowest it reaches, at 1,000 features, is 11.95. Bigger can be better; here it was not, and only the test told us.
Ages 8 to 11
Draw a wiggly line through every dot on a chart, then see how badly it guesses a new dot.
Ages 11 to 15
Split the Southwark table into training and test areas in Python and score a simple model.
Ages 15 and up
Code random features and the minimum-norm fit, plot both descents and add ridge.
Census inputs, our experiment
The area figures are Office for National Statistics Census 2021 data from Nomis, under the Open Government Licence. They describe output areas of roughly a hundred or more households, never a person or a home. The features, fits and error figures are our own, and a different random split would move them a little.
