The Leamington projectIterative proportional fitting: rebuilding a table from its edges
A real table with 860 cells, its inside hidden, and three attempts to put it back.
The census gives the type of home for every household in each of the 172 output areas of the Royal Leamington Spa built-up area. Summed by us, that is 22,096 households: 2,932 in detached houses, 6,348 semi-detached, 5,156 terraced, 5,602 in purpose-built flats and 2,058 in every other kind of home. Set out as a table of 172 areas by five types it has 860 cells. The learner keeps the 172 row totals and the five column totals, hides the cells, and asks a program to recover them.
The procedure needs a first guess, called the seed. With every cell set to 1, the program scales each row to its total, then each column to its total, and in this case it is finished after a single pass. Every total now matches to the household. Then the hidden table is uncovered and the two are compared cell by cell.
Rebuilding the Leamington housing table from its totals, our Python run| Seed | Passes needed | Households in the wrong cell | Cells within 10 of the truth |
|---|
| Every cell the same | 1 | 9,224 (41.7%) | 238 of 860 |
| Housing mix of the wider census zone | 2 | 8,569 (38.8%) | 298 of 860 |
| True figures of the closest other area | 5 | 7,994 (36.2%) | 416 of 860 |
All three rebuilt tables have perfect totals, and all three put more than a third of households in the wrong cell. One area of 236 households shows why. Its true row is 3 detached, 4 semi-detached, no terraced, 157 flats and 72 other. The flat-seed fit gives it 31, 68, 55, 60 and 22, which is simply the town average stretched to 236. The procedure cannot know that the area is mostly flats, because nothing in the totals says so. Better seeds helped a little: borrowing the housing mix of the wider zone cut the error to 38.8%, and copying the area next door cut it to 36.2%.
The copy-your-neighbour seed exposed a second trap. A cell that starts at zero is only ever multiplied, so it stays at zero for good. That seed held 86 zeros, and 53 of them were wrong: the true table has 694 households in those cells, and no amount of scaling could put them back. These results are for one town and one table; the method does better where rows resemble each other, and that is exactly what cannot be assumed.
Ages 8 to 11
A three by three grid with only the totals shown. Find one way to fill it in, then a second. Which is right?
Ages 11 to 15
Code the row step and the column step in Python for a small table and watch the totals settle.
Ages 15 and up
Run all three seeds on the 860-cell table, measure the error, and find the zeros that never recover.
Provenance
Housing counts are Census 2021 table TS044 from Nomis, published by the Office for National Statistics under the Open Government Licence. The procedure is from Deming and Stephan (1940). The choice of seeds, the error measure and the results are our own.
