Table of Contents
You type a few words, press Enter, and in well under a second a search engine picks the most useful pages out of billions. It feels like magic, but the core ideas are surprisingly approachable, and you can build a working miniature version in about forty lines of Python.
That is what this guide does. We build a small search engine over this very blog (it had 165 posts when we ran the code in September 2026), rank results with a classic formula called TF-IDF, and then run PageRank, the algorithm behind Google's early success, on the real links between our posts. The results, including the embarrassing ones, show exactly why ranking is the hard part.
The four jobs of a search engine
- Crawling: programs called crawlers or spiders download a page, find its links, and follow them to more pages, again and again.
- Indexing: each page's words are recorded in a giant lookup table, so the engine never has to read the pages again at search time.
- Ranking: when you search, every page that matches is given a score, and the scores decide the order.
- Serving: the top results are shown, usually with a title and a short description taken from the page.
Step 1 and 2: crawl and index
Our "crawl" is simple: instead of downloading the web, we read the title and description of every post on this blog. Then we build an inverted index: for every word, a list of the posts that contain it and how often. It is called inverted because it flips the pages around: instead of page to words, it goes from word to pages, exactly like the index at the back of a book.
import json, math, re, pathlib
from collections import defaultdict, Counter
# 1. "Crawl": read every blog post on this site (title and description)
docs = {}
for f in sorted(pathlib.Path("content/blog/data").glob("*.json")):
meta = json.loads(f.read_text(encoding="utf-8"))["meta"]
docs[meta["slug"]] = meta["title"] + " " + meta["description"]
STOP = {"a", "an", "and", "the", "of", "to", "in", "for", "on", "with", "is", "how", "what", "your", "you", "it", "from", "by", "or", "are", "at"}
def words(text):
return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
# 2. Index: for every word, which documents contain it, and how often
index = defaultdict(dict)
for slug, text in docs.items():
for word, count in Counter(words(text)).items():
index[word][slug] = count
print(f"indexed {len(docs)} posts, {len(index):,} distinct words")
print("posts containing 'fractions':", len(index["fractions"]))
print("posts containing 'python': ", len(index["python"]))
# 3. Rank: TF-IDF, so rare words count for more than common ones
def search(query, top=3):
scores = Counter()
for word in words(query):
postings = index.get(word, {})
if not postings:
continue
idf = math.log(len(docs) / len(postings))
for slug, tf in postings.items():
scores[slug] += tf * idf
return scores.most_common(top)
for w in words("fractions for children"):
print(f"idf of {w!r}: log({len(docs)} / {len(index[w])}) = {math.log(len(docs) / len(index[w])):.2f}")
for q in ["fractions for children", "python projects beginners", "olympiad proof"]:
print(f"\nsearch: {q!r}")
for slug, score in search(q):
print(f" {score:5.2f} {slug}")indexed 165 posts, 1,278 distinct words
posts containing 'fractions': 1
posts containing 'python': 42
idf of 'fractions': log(165 / 1) = 5.11
idf of 'children': log(165 / 6) = 3.31
search: 'fractions for children'
10.21 how-to-help-a-child-understand-fractions
3.31 how-coding-improves-mathematical-thinking-children
3.31 how-to-make-maths-fun-for-kids
search: 'python projects beginners'
9.08 30-plus-scratch-project-ideas-kids-fun-coding-beginner-advanced
9.04 python-for-beginners
7.67 file-built-in-methods-python-guide
search: 'olympiad proof'
8.83 pigeonhole-principle-explained
8.01 how-to-write-a-maths-proof
8.01 pythagoras-theorem-explained
The index holds 1,278 distinct words across 165 posts. "fractions" appears in just 1 post, while "python" appears in 42. Common words such as "the" and "how" are dropped as stop words, because they appear everywhere and help with nothing.
Step 3: rank with TF-IDF
Finding matching pages is easy. Putting them in the right order is hard. A classic starting point is TF-IDF: term frequency times inverse document frequency. A page scores higher when it uses a query word often (TF), and each word counts for more when it is rare across all pages (IDF). A word in every page tells you nothing; a word in one page tells you a lot.
For "fractions for children", the top result was our guide to helping a child understand fractions, with a score of 10.21, well clear of the rest. "fractions" appears in only 1 post, so its IDF is 5.11, while "children" appears in 6 posts and scores 3.31. That is a good result.
Where the tiny search engine goes wrong
Now look at "python projects beginners". The top result was 30-plus-scratch-project-ideas-kids-fun-coding-beginner-advanced, a Scratch post, just ahead of our Python beginners guide. Why? Our engine matches exact words only. It does not know that "project" and "projects" are the same word, that Scratch is not Python, or what the searcher actually wants. It simply added up scores for the words it could match.
- Word forms: real engines treat "project", "projects" and "projecting" as related, a technique called stemming.
- Meaning: modern engines use language models to understand that "learn to code" and "programming for beginners" mean nearly the same thing.
- Quality and trust: a page that matches the words is not necessarily good. That is where links come in.
PageRank: the idea that changed search
In 1998 Larry Page and Sergey Brin, then students at Stanford, described a way to measure a page's importance from the links pointing to it. The idea: a link is a vote, but votes from important pages count for more, and a page that links to many others spreads its vote thinly. Imagine a surfer clicking random links forever; PageRank is roughly the share of time they would spend on each page. We ran it on the real links between posts on this blog, as they stood in September 2026:
import json, re, pathlib
# Build the link graph: which blog posts link to which other blog posts
posts, links = {}, {}
for f in sorted(pathlib.Path("content/blog/data").glob("*.json")):
data = json.loads(f.read_text(encoding="utf-8"))
slug = data["meta"]["slug"]
posts[slug] = data["meta"]["title"]
text = json.dumps(data["content"]["sections"])
links[slug] = set(re.findall(r"/blog/([a-z0-9-]+)", text))
for slug in links:
links[slug] = {t for t in links[slug] if t in posts and t != slug}
# PageRank: a page is important if important pages link to it
n, d = len(posts), 0.85
rank = {s: 1 / n for s in posts}
for _ in range(50):
new = {s: (1 - d) / n for s in posts}
for s, outs in links.items():
if outs:
share = d * rank[s] / len(outs)
for t in outs:
new[t] += share
else: # a page with no links shares evenly
for t in posts:
new[t] += d * rank[s] / n
rank = new
inbound = {s: sum(s in outs for outs in links.values()) for s in posts}
total_links = sum(len(o) for o in links.values())
print(f"{n} posts, {total_links:,} internal links between them")
print("top 5 by PageRank:")
for s in sorted(rank, key=rank.get, reverse=True)[:5]:
print(f" {rank[s] * 1000:5.2f} {inbound[s]:>3} inbound {s}")165 posts, 385 internal links between them
top 5 by PageRank:
51.77 6 inbound usaco-bronze-to-silver-what-blocks-most-students
48.31 5 inbound big-o-notation-explained-simply
33.72 18 inbound python-for-beginners
31.56 5 inbound how-to-use-chatgpt-to-study-without-cheating
28.09 1 inbound how-to-teach-kids-ai-at-home
Look at the inbound counts. python-for-beginners has 18 posts linking to it, the most in the top five. Yet how-to-teach-kids-ai-at-home makes the top five with just 1. Its one link comes from a page that is itself highly ranked and links to very few other posts, so it passes on a large share of its importance. That is PageRank's key insight: the quality of links matters more than the quantity.
How modern search differs
Today's search engines use many signals beyond words and links, including how fresh a page is, whether it works well on a phone, and language models that interpret the meaning of a query. Google's own guide to its ranking systems says PageRank has evolved a lot since launch and remains part of its core ranking systems, alongside many others. The core pipeline of crawl, index and rank is still the same.
Why this matters for you
- Searching better: knowing that rare, specific words carry the most weight helps you write sharper searches. "Fractions for 8 year olds" beats "help with maths".
- Judging results: the top result is the one a formula scored highest, not a guarantee of truth.
- Learning to code: a search engine uses dictionaries, loops, sorting and a little logarithm maths. It is a great project once you know the basics, and it connects naturally to data structures and algorithms.
Finding the pages that match is easy. Deciding which one you actually wanted is the whole game.
How we teach it
A search engine is a good example of learning by building, one of the principles on our how we teach page: forty lines of Python teach dictionaries, loops and sorting in a way no worksheet can. Tracing the code line by line until every step can be predicted is how a result like the Scratch post topping a Python search stops being a mystery. Our Python course for teens runs one to one or in small groups of 5 to 10.
Frequently asked questions
They do four jobs: crawl the web by following links, index which words appear on which pages, rank the matching pages when you search, and show the best ones first. Crawling and indexing happen in advance, so searching is fast.
It is a lookup table from each word to the list of pages that contain it, like the index at the back of a book. It lets a search engine find every page with a word instantly instead of reading every page.
Term frequency times inverse document frequency. A page scores higher for a word it uses often, and each word counts for more if it is rare across all pages. It is a classic way to rank text results.
PageRank measures a page's importance from the links pointing to it. Links from important pages count for more, and a page with many outgoing links passes on less to each. It was described by Larry Page and Sergey Brin in 1998.
Yes, in an evolved form. Google's guide to its ranking systems says PageRank has changed a lot since Google launched and continues to be part of its core ranking systems, alongside many other systems.
Ranking is a prediction of what you want, based on words, links and other signals. A page can match your words without answering your question, and a simple engine that matches exact words, like ours, can be fooled easily.
Yes, a small one. With Python dictionaries you can build an inverted index and rank results with TF-IDF in about forty lines, as this post shows. It is an excellent intermediate coding project.
