Table of Contents
Every day, spam filters quietly move enormous numbers of junk messages out of inboxes without anyone noticing. They are one of the oldest and most successful uses of machine learning, and the classic method behind them is simple enough to build from scratch in about forty lines of Python, using nothing but word counts and a little probability.
This guide does exactly that. We train a small filter on example messages, test it on messages it has never seen, and look at what it learned, including one genuine mistake it makes and why. Along the way you will meet three ideas that apply to almost every machine learning system: learning from examples, the zero-probability trap, and the trade-off between false alarms and misses.
The idea: learn which words give spam away
Look at a few spam messages and patterns jump out: "free", "click", "claim", "winner", "now". Normal messages talk about lunch, homework and football. A spam filter turns that intuition into numbers. For every word, it learns how often it appears in spam and how often in normal messages. For a new message, it combines the evidence from all its words into one probability.
The method is called Naive Bayes, after Thomas Bayes, whose rule for updating probabilities with new evidence it uses. It is called naive because it treats every word as independent, ignoring word order and context. That is obviously not true of language, yet it works surprisingly well. An influential 2002 essay by the programmer Paul Graham, A Plan for Spam, did much to popularise this approach for email.
Step 1: examples to learn from
Machine learning starts with labelled examples. Here are 24 short training messages, 12 spam and 12 normal, plus 8 test messages kept separate so we can check the filter honestly. They were written for this example; real filters learn from millions of real messages.
# Short messages written for this example. 1 = spam, 0 = normal ("ham").
train = [
("Congratulations you have won a free prize claim it now", 1),
("You are a winner click here to claim your cash prize", 1),
("Free gift card waiting click the link now", 1),
("Urgent your account is locked click here to verify now", 1),
("Claim your free holiday winner selected today", 1),
("Earn cash fast from home click now limited offer", 1),
("Limited offer free phone for the first 100 winners", 1),
("Your parcel is waiting click the link to pay the fee", 1),
("Act now to claim your reward before it expires", 1),
("Hot deal free trial click here today only", 1),
("Verify your password now or your account will close", 1),
("You won cash click to collect your prize today", 1),
("Are we still meeting for lunch tomorrow", 0),
("Can you send me the notes from maths class", 0),
("Mum says dinner is at seven tonight", 0),
("The coding club meets in room 4 after school", 0),
("Thanks for the help with my homework yesterday", 0),
("Do not forget your football kit tomorrow", 0),
("I finished the Python project, want to see it", 0),
("Happy birthday, hope you have a great day", 0),
("The bus is late, I will be there in ten minutes", 0),
("Can you check my essay before I submit it", 0),
("Our team won the match today", 0),
("Please bring the library book back on Monday", 0),
]
test = [
("Click here now to claim your free prize", 1),
("Winner selected claim your cash today", 1),
("Your account will close, verify your password here", 1),
("Limited free offer for winners click now", 1),
("Can we meet after school to finish the project", 0),
("See you at football practice tomorrow", 0),
("I won the maths quiz today", 0),
("Dinner is ready, come home now", 0),
]Step 2: train and test the filter
Training is just counting. Classifying multiplies the chances of each word appearing in spam, and compares with the same for normal messages. To avoid numbers so small the computer rounds them to zero, the code adds logarithms instead of multiplying, which gives the same answer safely.
import math, re
from collections import Counter
from messages import train, test
def words(text):
return re.findall(r"[a-z]+", text.lower())
# Training: count how often each word appears in spam and in normal messages
counts = {1: Counter(), 0: Counter()}
docs = Counter()
for text, label in train:
counts[label].update(words(text))
docs[label] += 1
vocab = set(counts[1]) | set(counts[0])
def p_word(word, label):
# add-one smoothing: a word never seen in this class still gets a small chance
return (counts[label][word] + 1) / (sum(counts[label].values()) + len(vocab))
def spam_probability(text):
log_score = {}
for label in (1, 0):
log_score[label] = math.log(docs[label] / len(train))
for w in words(text):
log_score[label] += math.log(p_word(w, label))
# turn the two scores back into a probability between 0 and 1
top = max(log_score.values())
s, h = math.exp(log_score[1] - top), math.exp(log_score[0] - top)
return s / (s + h)
if __name__ == "__main__":
print(f"trained on {len(train)} messages, {len(vocab)} different words")
spammy = sorted(vocab, key=lambda w: p_word(w, 1) / p_word(w, 0), reverse=True)[:6]
print("most spam-like words:", ", ".join(spammy))
correct = 0
for text, label in test:
p = spam_probability(text)
guess = 1 if p > 0.5 else 0
correct += guess == label
print(f"{p:6.3f} {'SPAM' if guess else 'ok '} {'right' if guess == label else 'WRONG'} {text}")
print(f"test accuracy: {correct} of {len(test)}")trained on 24 messages, 121 different words
most spam-like words: click, now, free, claim, your, prize
1.000 SPAM right Click here now to claim your free prize
0.999 SPAM right Winner selected claim your cash today
0.999 SPAM right Your account will close, verify your password here
1.000 SPAM right Limited free offer for winners click now
0.024 ok right Can we meet after school to finish the project
0.029 ok right See you at football practice tomorrow
0.145 ok right I won the maths quiz today
0.835 SPAM WRONG Dinner is ready, come home now
test accuracy: 7 of 8
From just 24 examples the filter picked out click, now, free, claim, your as the strongest signs of spam. Nobody told it; it counted. On the 8 test messages it got 7 of 8 right.
The mistake, and what it teaches
The filter marked "Dinner is ready, come home now" as spam with a probability of 0.83. Why? The word "now" appeared in many spam training messages ("click now", "act now") and never in a normal one. The filter has no idea that "come home now" is a perfectly ordinary thing to say. It only knows word counts.
This is how machine learning goes wrong in general: the model learns patterns from its training data, including accidental ones. More varied training data would fix this particular mistake, because real normal messages use "now" all the time. Our post on why machine learning models make mistakes explores this with more experiments.
The zero-probability trap
One detail in the code matters more than it looks: the + 1 in p_word. Without it, a word that never appeared in spam during training gets a probability of exactly zero, and multiplying by zero wipes out every other piece of evidence:
import re
from collections import Counter
from messages import train
counts = {1: Counter(), 0: Counter()}
for text, label in train:
counts[label].update(re.findall(r"[a-z]+", text.lower()))
vocab = set(counts[1]) | set(counts[0])
def p_raw(word, label): # no smoothing
return counts[label][word] / sum(counts[label].values())
def p_smooth(word, label): # add-one smoothing
return (counts[label][word] + 1) / (sum(counts[label].values()) + len(vocab))
message = "click here to claim your free prize for your homework"
for name, p in [("without smoothing", p_raw), ("with smoothing", p_smooth)]:
spam = ham = 0.5
for w in message.split():
spam *= p(w, 1)
ham *= p(w, 0)
verdict = "SPAM" if spam > ham else "ok"
print(f"{name:<18} spam score {spam:.3g} normal score {ham:.3g} -> {verdict}")
print("'homework' seen in spam:", counts[1]["homework"], "times")without smoothing spam score 0 normal score 0 -> ok
with smoothing spam score 4.5e-18 normal score 9.9e-23 -> SPAM
'homework' seen in spam: 0 times
Without smoothing, both scores collapse to zero and the filter cannot decide, so a spam message slips through just by mentioning homework. Adding one imaginary sighting of every word to each class, called add-one or Laplace smoothing, fixes it. The same trick appears throughout machine learning whenever a model meets something it has never seen.
False alarms versus misses
A filter outputs a probability, but it still has to decide: above what value do we call it spam? That choice is a trade-off:
from bayes import spam_probability
from messages import test
for threshold in (0.5, 0.9, 0.999):
false_alarms = sum(spam_probability(t) > threshold and label == 0 for t, label in test)
missed = sum(spam_probability(t) <= threshold and label == 1 for t, label in test)
print(f"threshold {threshold}: {false_alarms} normal message(s) flagged, {missed} spam missed")threshold 0.5: 1 normal message(s) flagged, 0 spam missed
threshold 0.9: 0 normal message(s) flagged, 0 spam missed
threshold 0.999: 0 normal message(s) flagged, 1 spam missed
At 0.5 the dinner message is wrongly flagged. At 0.9 everything is right on this small test set. Push the bar to 0.999 and a real spam message slips through. Real email services lean towards caution, because a lost job offer or school message costs far more than one extra junk email. It is the same false alarm problem that makes AI detectors unreliable, and it comes straight from basic probability.
How real spam filters differ
Modern email services use much larger machine learning models, trained on huge numbers of messages, and combine the words with other signals such as the sender's reputation, links in the message and how other users have reported similar mail. The ideas in this post, learning from labelled examples and balancing false alarms against misses, still sit at the core.
Try it yourself
- Copy the three files and run
bayes.py. - Add a few normal messages that use "now" and retrain. Does the dinner message stop being flagged?
- Write a spam message designed to fool the filter, using only normal-looking words. What does that tell you about spammers?
- Print the 6 most normal words, the ones most likely in normal messages.
A spam filter does not understand language. It counts, and it is very good at counting.
How we teach it
A spam filter is a good example of two principles on our how we teach page. Learning by building: forty lines of code teach more about machine learning than a page of definitions. And mistakes are data, not failures, which is exactly how the dinner-message false alarm should be read. Our AI and machine learning course for teens runs one to one or in small groups of 5 to 10.
Frequently asked questions
They learn from labelled examples which words and features are common in spam and which are common in normal messages. For a new message, they combine the evidence into a probability of spam and move it to the spam folder if that probability is above a threshold.
It is a classic filter that uses Bayes' rule to combine the probability of each word appearing in spam and in normal messages. It is called naive because it treats every word as independent, ignoring order and context, yet it works well in practice.
Because the filter judges patterns, not meaning. A normal message that happens to use words common in spam, like our example with the word now, can get a high spam score. That is called a false positive.
It adds one imaginary count of every word to each class, so no word ever has a probability of exactly zero. Without it, one word the filter has not seen in spam would wipe out all the other evidence.
Yes, a spam filter is a classic example of machine learning, a branch of AI where a system learns patterns from examples rather than following rules written by hand.
Yes. A simple Naive Bayes filter needs only word counts, a dictionary and some probability, about forty lines of Python, as shown in this post. It is a good first machine learning project.
Spammers constantly change their wording, so fixed rules go out of date quickly. A probability-based filter can be retrained on new examples and weighs many clues together instead of relying on any single word.
