The Halesowen projectAn agent that admits doubt: conformal prediction intervals and abstention
Predict, attach an honest range, and decline the questions you are least sure of.
Three Census 2021 tables, pulled through the Nomis API, cover every one of Dudley borough's 1,033 output areas. Of 137,044 households, 41,701 are one person living alone, and the agent's job is to predict that share for each area from two clues: how densely populated it is and how many households have no car. A random forest does the predicting. The areas are split three ways: 60% to train on, 20% held back as a calibration set, and 20% as a final test, and the whole experiment is repeated on 50 different random splits.
First the agent needs a range it can stand behind, a prediction interval. The tempting shortcut is to look at its errors on the training areas and take a range wide enough to cover 90% of them. Split conformal prediction does the same thing on the calibration set instead, areas the model never trained on, with a small correction for the size of that set.
Promised 90% ranges for each area's one-person share, tested on unseen areas, averages of 50 splits, our Python run on Census 2021 data| How the range was set | Range either side of the prediction | Test areas actually inside it |
|---|
| From errors on training areas | 9.34 points | 76.0% |
| Conformal, from the calibration set | 15.31 points | 90.6% |
The shortcut promised 90% and delivered 76.0%, because a model always looks better on data it has already seen. The conformal range is wider and honest, landing at 90.6% on average (between 84.5% and 96.6% on individual splits, as expected with about 200 test areas). Next comes abstention. The forest is 300 trees, and where they disagree most the agent is least sure, so it can answer only its most confident cases and pass the rest to a person.
Average error when the agent answers only its most confident share of test areas, our simulation| Share of areas answered | Average error (points) |
|---|
| All of them | 6.66 |
| Most confident 80% | 5.97 |
| Most confident 60% | 5.26 |
| Most confident 40% | 4.74 |
| Most confident 20% | 4.43 |
Saying "pass" on the doubtful 60% cuts the average error from 6.66 to 4.74 points, against 8.86 for simply guessing the borough average every time. The price is that a person must handle the rest. Choosing that trade-off is a design decision, not something the agent should decide for itself.
Ages 8 to 11
Guess sweets in a jar with a range instead of one number, and check how often the range catches the answer.
Ages 11 to 15
Predict one-person shares for Dudley areas in Python and count how often a stated range is right.
Ages 15 and up
Build split conformal intervals, test coverage and design an agent that knows when to abstain.
Census data, our agent
Household, car and density counts are Office for National Statistics Census 2021 data from Nomis, under the Open Government Licence. The model, the ranges, the abstention rule and every figure in the tables are our own work.
