The Southport projectHow ChatGPT-style tokenizers learn: byte-pair encoding on England's town names
Train a tokenizer on 5,124 names, test it on 1,282 it has never seen, and read what its tokens do and do not capture.
The learner takes the ONS list of built-up areas in England from the 2021 census, removes the bracketed district labels, and splits each name into words. Every word starts as a row of single letters with an end-of-word marker, written here as an underscore. Byte-pair encoding then counts every pair of neighbouring symbols across all the names, glues the most common pair into a new single symbol, and repeats. Each glue step is called a merge, and the list of merges is the tokenizer. Four names in five are used for training and the rest are kept back for testing.
The first merges read like a lesson in English place names: "n" at the end of a word, then "on" at the end, then "ton" at the end, all within the first three steps. That is no accident. Of the 6,406 names, 1,049 end in "ton", far more than any other ending we counted: "ham" ends 354, "ley" 309, "ford" 201, "field" 147 and "port" just 14. Soon the tokenizer has whole endings such as "ham", "ley", "ford", "ington" and "ston" as single tokens.
How many tokens an unseen English place name needs, on average, after each number of merges, our Python run, 29 September 2026| Merges learned | Tokens per unseen name |
|---|
| 0 (single letters and word ends) | 11.39 |
| 50 | 7.01 |
| 200 | 5.00 |
| 500 | 3.95 |
Local names cut into tokens after 200 and after 500 merges (| separates tokens, _ ends a word)| Name | 200 merges | 500 merges |
|---|
| Southport | sou|th|p|or|t_ | south|port_ |
| Birkdale | bi|r|k|d|al|e_ | bir|k|dale_ |
| Ainsdale | a|in|s|d|al|e_ | a|in|s|dale_ |
| Churchtown | ch|ur|ch|t|own_ | ch|ur|ch|town_ |
More merges mean fewer, bigger tokens: an unseen name shrinks from 11.39 symbols to 3.95 tokens, which is why tokenizers exist, since shorter sequences are cheaper for a model to process. And because every single letter stays in the vocabulary, any name can still be written down, even one the tokenizer has never met. But look at the pieces. After 500 merges, Southport splits into "south" and "port", which happens to match its meaning. Ainsdale becomes "a", "in", "s" and "dale", and Churchtown "ch", "ur", "ch" and "town". The tokenizer knows nothing about meaning; it only knows which letters are common together, and a rarer word simply gets cut into more pieces.
Ages 8 to 11
Collect local street and town names and hunt for the endings that repeat, like "ton" and "dale".
Ages 11 to 15
Count letter pairs across a list of names in Python and make the first few merges by hand.
Ages 15 and up
Write the full byte-pair encoder, test it on unseen names and measure tokens per name.
ONS names, our tokenizer
The place names are from the Office for National Statistics 2021 census built-up area tables. The tokenizer, the merges, the counts and the examples are our own work, and real AI companies' tokenizers are trained on far more text.
