The Scunthorpe projectChecking an AI agent's tool output: real traffic data from the DfT
Fetch the records, write down what you expect them to look like, and count every place the data disagrees.
The learner writes a small Python "tool" of the kind an AI agent would call: it asks the Department for Transport's road traffic API for every average daily flow record in North Lincolnshire, following the pages politely. Eight requests return 1,844 records covering 154 counting points from 2000 to 2025. One of them is the A18 between the Scotter Road roundabout and the A159, where 15,374 motor vehicles a day were counted in 2024. Before answering any question, the learner writes a schema: a description of what each field should be, such as "latitude is a number", "the vehicle types add up to the total" and "every record says whether it was counted or estimated". Then every record is checked.
What the schema checks found in 1,844 DfT records for North Lincolnshire, our Python run, 29 September 2026| Check | Result |
|---|
| Latitude and longitude supplied as numbers | No: text in all 1,844 records |
| Road-link length and junction names present | Missing in 436 records |
| Vehicle types add up to the all-vehicles total | Off in 462 records, by one or two vehicles |
| HGV classes add up to the HGV total | Off in 464 records |
| Records estimated rather than counted | 1,078 of 1,844 |
| Counting points never counted at all | 5 of 154 |
Each finding is a trap for an agent. Because the coordinates arrive as text, adding two of them with a plus sign in Python joins them into nonsense, "53.5494299853.54942998", instead of adding the numbers; the fix is to convert types deliberately, straight after the tool call. Missing link lengths mean any "traffic per kilometre" answer has to leave out almost a quarter of the records, or say that it cannot answer. The totals that are one or two vehicles out look like mistakes but are consistent with each figure being rounded separately, so a sensible agent reports the published total rather than "correcting" it by adding up the parts.
The last check matters most for honesty. The field "estimation_method" shows that 1,078 records, 58%, are estimates rather than counts, and the most common detail is "Estimated using AADF from previous year on this link", which appears 1,050 times. In 2025, 23 records were counted and 44 estimated. An agent that reports "the A18 carries 15,374 vehicles a day" should add that this was a counted figure; for most other roads, the same kind of number is an estimate built from an earlier year.
Ages 8 to 11
Check a printed table for gaps and odd entries, and explain what each problem could lead to.
Ages 11 to 15
Fetch the traffic records in Python, convert the text numbers and count the missing values.
Ages 15 and up
Write a full schema, validate every record and design how an agent should report each problem.
DfT data, our checks
Traffic records come from the Department for Transport road traffic statistics service, and the census counts from the ONS. The schema, the checks and every count in the table are our own work; none of the findings is a criticism of the publisher.
