The Hoddesdon projectElo ratings: ranking 120 areas from noisy head-to-head judgements
A true answer the learner can check, a simulated judge who makes mistakes, and a rating that updates after every verdict.
Census table TS006 gives the population density of every output area. Across the 120 areas of Hoddesdon in Broxbourne it runs from 114 to 17,778 residents per square kilometre, with a median of 5,592. That list is the truth, and the rating system never sees it. Instead a simulated judge is shown two areas and says which is denser. The judge is almost always right when the difference is large and close to guessing when two areas are similar; over 19,827 random pairs it was right 78.3 per cent of the time.
Every area starts on 1,000 points. Before each verdict the program works out how likely each side was to win from the gap in their ratings; afterwards the winner gains, and the loser drops, the same amount: a fixed number K times the difference between what happened and what was expected. A large K makes ratings jump quickly; a small K makes them steady but slow. In Python the whole update is three lines inside a loop.
How well Elo recovers the true order, average of 30 runs of random pairs, our simulation| Judgements | Rank match, K 16 | True top 10 found, K 16 | Rank match, K 64 | True top 10 found, K 64 |
|---|
| 600 | 0.81 | 4.0 | 0.82 | 4.2 |
| 1,200 | 0.89 | 4.5 | 0.89 | 4.6 |
| 2,400 | 0.93 | 6.0 | 0.92 | 5.7 |
| 4,800 | 0.96 | 7.3 | 0.93 | 6.3 |
Rank match here is the Spearman correlation between the ratings and the true densities, where 1 would be perfect. More judgements helped steadily: with 4,800, about 80 appearances per area, a steady K of 16 found 7.3 of the true top 10 on average. A jumpy K of 64 kept pace at first but fell behind later, because each fresh result kept knocking the ratings around.
The learner then takes one fixed set of 2,400 judgements and replays them in 100 different orders. The verdicts are identical; only the sequence changes. With K 16 the typical area moved 13.5 places between its highest and lowest finish, and the same area finished top every time. With K 64 the typical area moved 41 places and 11 different areas finished top. Elo remembers recent results more than old ones, so the answer depends on when things happened, not only on what happened. The final test compares areas only with near neighbours in the true order, the way many online games try to pair players of similar strength. After 4,800 such judgements the rank match was about zero: when every contest is between near-equals, the verdicts are close to coin tosses and carry almost no information.
Ages 8 to 11
Rank six mystery jars by weight using only a balance, two at a time, and count the weighings needed.
Ages 11 to 15
Code the Elo update in Python, run a league of eight teams and watch the table change after each match.
Ages 15 and up
Simulate the noisy judge, sweep K and the number of judgements, and measure how order changes the result.
Sources and limits
Densities are Census 2021 table TS006 from Nomis, under the Open Government Licence. The judge and all verdicts are simulated by us, not collected from people, and every figure is from our own runs, which is why they are averages over repeats. Chatbot Arena is described in a 2023 post by LMSYS.
