The Smethwick projectDistance concentration: when every area looks equally far away
Describe 978 areas with more and more columns, and compare each area's nearest match with its farthest.
From the Nomis API the learner pulls four Census 2021 tables for all 978 output areas in Sandwell: household size, car or van availability, accommodation type and population density. The household-size rows add to 130,315 households (the published borough total is 130,246; small census counts are adjusted, so rows and totals differ a little). Turned into shares, the tables give 21 columns per area, each rescaled so no column dominates. For every area the program finds the closest other area and the most distant one, then divides the first distance by the second. A ratio close to 0 means the nearest match is far nearer than the rest, which is what a similarity search needs. A ratio close to 1 means near and far are almost the same.
Nearest distance as a fraction of farthest distance, median across 978 areas, our Python run| Columns used | Sandwell census columns | Independent random columns |
|---|
| 3 | 0.01 | 0.04 |
| 8 | 0.06 | 0.20 |
| 21 | 0.14 | 0.40 |
| 100 | not available | 0.67 |
| 1,000 | not available | 0.88 |
Both columns of results rise, so the effect is real in the census data too: with all 21 columns the nearest area is 0.14 of the way to the farthest, against 0.01 with three. The random points are much worse, reaching 0.88 at 1,000 columns, where "nearest" has almost no meaning. The census data resists because its columns overlap. An area with many flats tends to have fewer cars, so the 21 columns carry less than 21 separate pieces of information; one standard measure, the participation ratio, puts it at 8.8. Counting columns is not the same as counting information.
The second experiment is harsher. Using three sensible columns (density, the share of homes with no car, the share of one-person homes), the program records each area's nearest neighbour. It then adds three columns of pure random noise and looks again. Only 4.8% of areas keep the same nearest neighbour, and with ten noise columns 0.9% do. A few meaningless columns are enough to scramble the matches. Figures for fewer than 21 census columns are medians over 30 random choices of columns, and the noise results are medians over 10 random seeds.
Ages 8 to 11
Sort picture cards by one feature, then by five at once, and see how "most alike" gets harder to agree on.
Ages 11 to 15
Write a Python distance function and find the closest match for a Smethwick area on two columns, then on ten.
Ages 15 and up
Reproduce the ratio table with NumPy, add noise columns and explain why correlated columns behave differently.
Whose numbers these are
The census counts are Office for National Statistics data served by Nomis under the Open Government Licence. Ratios, medians and the noise experiment are our own calculations, and the random columns are generated by us, not measured anywhere. Areas are compared only on housing, household size, cars and density.
