The Dewsbury projectA language model built from the Life of Charlotte Bronte
Count word pairs in one volume, predict and generate text, then test on the other volume.
The learner downloads both volumes, lowercases the words and marks where each sentence starts. The first run gives a strange result: after the word "mr", the model predicts a new sentence every single time, 208 out of 208. The cause is the full stop in "Mr.", which the program read as the end of a sentence. Once titles like Mr, Mrs and Dr are handled, "mr" is followed by "bronte" 55 times, the most common pair. Volume 1 then yields 98,792 tokens and 54,305 different word pairs. Some predictions are perfect: "roe" is followed by "head" all 24 times, and "dewsbury" by "moor" 9 times out of 11.
What the bigram model learned from Volume 1, and how it did on Volume 2, our Python run, 28 September 2026| Test | Result |
|---|
| Word after "roe" | "head", 24 times out of 24 |
| Word after "dewsbury" | "moor", 9 times out of 11 |
| Greedy text from "she" | "she was a little more than i have been a little more than i have" |
| Word pairs in Volume 2 never seen in Volume 1 | 40,656 of 89,743 (45.3%) |
| Perplexity on Volume 2, word frequency only | 824.6 |
| Perplexity on Volume 2, bigram with add-one smoothing | 3,819.6 |
| Perplexity on Volume 2, half bigram and half word frequency | 483.0 |
Asking the model to write shows its first weakness. Always choosing the single most likely next word, known as greedy decoding, traps it in a loop: "she was a little more than i have been a little more than i have". Chatbots usually add some controlled randomness when choosing words, partly for this reason. The second weakness appears on the held-out volume. Nearly half of the word pairs in Volume 2, 45.3%, never occur in Volume 1, so a raw bigram model would give those sentences a probability of zero, which is plainly wrong. The standard fix, pretending every pair was seen once more than it was, turns out to be too crude here.
Perplexity measures how surprised a model is by new text, roughly how many words it is choosing between at each step, so lower is better. A model that ignores context and just uses word frequencies scores 824.6 on Volume 2. The bigram model with add-one smoothing does far worse, 3,819.6, because spreading probability over nearly 13,000 possible words drowns out what it learned. Mixing the two, half bigram and half word frequency, brings it down to 483.0, better than either alone. The mixing weight was tried on Volume 2 itself, a shortcut a careful learner would replace with a separate test slice.
Ages 8 to 11
Play "what word comes next?" with a favourite story and tally the guesses.
Ages 11 to 15
Count word pairs in Python and generate sentences from the counts.
Ages 15 and up
Test on held-out text, measure perplexity and compare smoothing methods.
Gaskell's words, our model
The text is the Project Gutenberg edition of The Life of Charlotte Bronte by Elizabeth Cleghorn Gaskell. The counting, the model and every figure in the tables are our own work.
