The Nuneaton projectA retrieval AI agent meets the Liggins hoax
Split a book into passages, let the agent search them, and check what it answers.
Many AI agents answer questions by first searching a set of documents and then writing an answer from what they find, a method called retrieval-augmented generation. The learner vibe codes a small version. Blind's biography is split into 354 passages of at least twenty words. A standard scoring method called BM25 ranks the passages for each question by how often its rarer words appear. An answer step then reads the top passage. Asked "Which school did Mary Ann go to in Nuneaton?", the agent finds the right passage first time: "a school at Nuneaton, kept by Miss Lewis." It looks like it works.
What the retrieval agent found in Mathilde Blind's George Eliot (1883), our Python run, 28 September 2026| Question | Where the right passage ranked | What went wrong |
|---|
| Which school did Mary Ann go to in Nuneaton? | 1st | Nothing: the answer is Miss Lewis's school |
| Who wrote Adam Bede? | 1st, but it is the Liggins passage | A naive answer picks "Liggins", named 7 times there |
| Where was George Eliot born? | 3rd | A quotation containing "born" ranked higher |
| Same question, common words kept | 11th | Words like "where" and "was" swamped the search |
Then the learner asks "Who wrote Adam Bede?" The top passage is the one about the hoax, and it names Liggins seven times. A simple answer step that picks the most mentioned name replies "Liggins". The passage itself makes clear the claim was false: George Eliot challenged the pretender to produce a specimen of his writing in the style of Adam Bede. The retrieval was not wrong; the reading was. The agent treated a reported rumour as a fact, the same kind of mistake an AI assistant can make when it summarises a page that quotes someone else's false claim.
A second lesson comes from "Where was George Eliot born?" The right passage, "born on November 22nd, 1819, at South Farm, a mile from Griff", only ranks third, beaten by a quotation that happens to contain the word born. If the program keeps small words like "where" and "was" in the search, it falls to eleventh. Blind herself warns that "both the date and place of her birth have been incorrectly stated, hitherto", so sources disagree. The fix the learner builds is simple and powerful: the agent must show the passage it used for every answer, so a reader can check whether it states a fact or reports a claim.
Ages 8 to 11
Find answers in a short story, and spot sentences that say what someone believed rather than what was true.
Ages 11 to 15
Vibe code a keyword search over a book and test it with questions whose answers you know.
Ages 15 and up
Build BM25 retrieval, measure where the right passage ranks, and add citations to every answer.
Blind's book, our agent
All quotations come from the Project Gutenberg edition of Mathilde Blind's George Eliot (1883). The passage splitting, the ranking and every result in the table are our own work.
