The Bury St Edmunds projectMatrix completion: guessing the missing cells of a census table
141 areas, 11 ways of getting to work, and a fifth of the numbers deliberately removed.
Census table TS061 records how employed residents usually travel to work: from home, by train, bus, taxi, motorbike, car as driver or passenger, bicycle, on foot, or another way, plus a category for underground, metro, light rail and tram that is almost empty here. For the 141 output areas of the Bury St Edmunds built-up area that makes a table of 1,551 numbers, and 483 of them are zero. The learner turns each row into shares of that area's workers, then hides 20 per cent of the cells at random and tries to predict them.
The first guess is the column average: whatever share of workers use a method across the known cells, assume the same in the hidden one. The smarter guess is a low-rank model. It gives every area a short list of hidden numbers and every travel method a matching list, and predicts each cell by multiplying the two lists together. The learner fits the lists with alternating least squares: hold the method numbers fixed and solve for each area, then hold the areas fixed and solve for each method, and repeat. Each step is an ordinary least-squares fit, a few lines of NumPy.
Average miss per hidden cell, in people, 20 random masks with 20 per cent hidden, our Python run| Method | Hidden cells | Known cells | Beat the column average |
|---|
| Column average | 3.52 | 3.46 | not applicable |
| Rank 1, penalty 0.1 | 3.38 | 2.35 | 13 of 20 |
| Rank 2, penalty 0.1 | 2.31 | 1.19 | 20 of 20 |
| Rank 3, penalty 0.1 | 2.30 | 0.98 | 20 of 20 |
| Rank 10, penalty 0.001 | 3.57 | 0.02 | 7 of 20 |
Two hidden numbers per area were enough to cut the average miss from 3.52 people to 2.31, and the model won on every one of the 20 masks. The last row is the warning. With ten hidden numbers and almost no penalty for large values, the model fitted the cells it could see almost perfectly, missing by 0.02 people on average, and then did worse than the plain column average on the cells it could not see. It had memorised instead of learning. The penalty, called regularisation, is what stops that: with it set to 0.1, rank 10 missed by 2.40 people, close to rank 3.
Hiding half the table instead of a fifth made everything harder: rank 2 missed by 2.95 people against 3.51 for the column average, and still won in 19 of 20 masks. In three of those masks one area had every cell hidden, and no model can say anything about an area it has never seen except the average. Recommender systems call this the cold start problem, and a new customer with no history meets exactly the same wall.
Ages 8 to 11
A times table with blanks: fill them from the row and column numbers, then invent a table with no pattern and try again.
Ages 11 to 15
Predict hidden cells with column averages in Python and measure the miss in people, not percentages.
Ages 15 and up
Write alternating least squares, sweep the rank and the penalty, and plot error on known and hidden cells.
Sources and limits
Census 2021 table TS061 from Nomis, published by the ONS under the Open Government Licence. The low-rank method follows Koren, Bell and Volinsky (2009) and Candès and Recht (2009). Hidden cells were chosen at random by us, and each score is an average over 20 random choices, so another run will differ slightly.
