The Walsall projectWord embeddings from the Annals of Willenhall
Count which words appear together, turn the counts into vectors, then ask what sits nearest to "lock".
The learner strips the Project Gutenberg header and footer, lowercases the text, keeps only letters and drops single letters. That leaves 59,238 words, 8,224 of them different, of which 745 appear ten or more times; those 745 form the vocabulary. For every vocabulary word the program counts which other vocabulary words appear within four places of it. Each word is now a row of co-occurrence counts, a first, crude embedding, and two words can be compared by the angle between their rows, a measure called cosine similarity that runs from 0 for nothing in common up to 1.
Nearest words by raw co-occurrence counts, our Python run on the Annals of Willenhall, 28 September 2026| Word | Six nearest neighbours |
|---|
| lock | year, of, was, however, at, last |
| church | in, and, for, which, by, being |
| walsall | town, one, in, two, and |
The raw counts produce nonsense, and the reason is frequency. Words such as "the", "of" and "and" appear next to almost everything, so they dominate every row and make every word look like every other. The fix is a weighting called positive pointwise mutual information, PPMI for short, which asks whether two words appear together more often than their separate frequencies would predict, and keeps only the pairs that do. With PPMI weighting the neighbours change completely.
Nearest words after PPMI weighting, with cosine similarity for "lock", same text and run| Word | Nearest neighbours |
|---|
| lock | making 0.342, carpenter 0.279, young 0.262, keys 0.255, not 0.246, key 0.242 |
| locks | industry, key, trade, are, keys, various |
| iron | brass, founder, master, key, lock, patent |
| church | st, parish, collegiate, the, wolverhampton, willenhall |
Most of that looks sensible, yet two neighbours of "lock" deserve a second look. "Carpenter" is not about woodwork: it is James Carpenter, whose name appears within four words of "lock" five times in the book. "Young" is not about age either. Because the text was lowercased, the surname in "the Carpenter and Young invention" was merged with the ordinary word "young". The model has no way to tell a name from a meaning; it only knows which words keep company. A learner who spots this has found, in miniature, why AI systems sometimes link things for reasons nobody intended.
Test: how many of the five nearest neighbours of 13 town names are also town names (65 places)| Method | Town neighbours found | Of which "willenhall" or "wolverhampton" |
|---|
| Raw counts | 15 of 65 | 11 |
| PPMI weighting | 14 of 65 | 3 |
A quick score makes the two methods look almost equal, 15 against 14. Opening up the results tells a different story. In the book, "willenhall" is used 504 times and "wolverhampton" 216, far more than any other town on the list, so the raw method lists them next to almost every place name for the same reason it lists "the". Eleven of its fifteen hits are those two words. PPMI finds a wider mix of genuine town pairs, such as "darlaston" beside "wednesbury". Checking what sits behind a score, and not only the score, is the habit this project is built to teach.
Ages 8 to 11
Play odd one out with word cards, then explain in a sentence what the others share.
Ages 11 to 15
Count word neighbours in Python and list the most common company a word keeps.
Ages 15 and up
Build PPMI vectors, rank neighbours by cosine and design a fair test of the result.
Hackwood's text, our vectors
The book is the Project Gutenberg transcription of The Annals of Willenhall by Frederick Wm. Hackwood. The word counts, the embeddings, the similarity scores and the town test are our own work.
