The Kidderminster projectImportance sampling: an agent with twenty looks
A known total, a small budget, and two ways of choosing where to look.
The test bed is the Census. Wyre Forest has 338 output areas, and because the ONS has counted all of them we know the true totals: 101,617 residents by our sum of the areas, in 45,307 households. That lets us mark the agent's work. The agent is told the number of households in every area, which stands for the cheap information an agent usually has, and is allowed to "visit" only a few areas to learn how many residents each one holds. From those visits it must estimate the district total. It plays the game 5,000 times with each method so that its typical error can be measured.
The uniform agent picks areas with equal chance and scales up the average. The weighted agent picks areas with a chance proportional to their household count, so bigger areas are visited more often, and then divides each area's residents by the chance that area had of being chosen before averaging. That division is the whole trick: it cancels the favouritism exactly, so the estimate stays fair on average.
Typical error in the agent's estimate of Wyre Forest's residents, as a percentage of the true total, over 5,000 repeats (our Python run on Census 2021 data)| Areas visited | Equal chance | Weighted by households |
|---|
| 5 | 9.4% | 5.9% |
| 10 | 6.7% | 4.2% |
| 20 | 4.6% | 3.0% |
| 40 | 3.3% | 2.1% |
| 80 | 2.3% | 1.4% |
Twenty weighted visits beat forty unweighted ones. The agent has not worked harder; it has looked in better places and been honest about having done so. The method is only as good as the hunch behind it, though. When the agent must estimate the 4,464 households living in purpose-built flats, which 120 areas do not have at all, equal-chance sampling with 20 visits is off by 38.6% and weighting by households barely helps, at 35.9%, because large areas are not especially full of flats. Weighting by the number of one-person households, a better clue, brings it to 26.9%. And if the weights point the wrong way, favouring areas with few households, the error in the residents estimate rises to 9.1%, about three times the weighted result. In every one of these runs the average of the estimates stayed within half a percent of the truth: the correction keeps the agent unbiased even when its hunch is poor. What a poor hunch costs is reliability.
Ages 8 to 11
Estimate the sweets in ten jars by opening three, first at random, then choosing the big jars and adjusting.
Ages 11 to 15
Simulate both agents in Python on a list of 30 numbers and compare how far off each tends to be.
Ages 15 and up
Run 5,000 repeats on all 338 areas, try three different proxies and plot error against budget.
About the numbers
Resident, household and accommodation counts are Office for National Statistics Census 2021 data taken from Nomis under the Open Government Licence. The agents, the repeats and the error figures are ours. The 101,617 is our own total of 338 areas and is used only to score the agent; it is not an ONS headline figure.
