---
title: "How Do Search Engines Work? Build a Tiny One in Python"
description: "How search engines crawl, index and rank pages: build a tiny one in Python with TF-IDF, then run PageRank on this blog's real links."
slug: how-do-search-engines-work
canonical: https://learn.modernagecoders.com/blog/how-do-search-engines-work/
date: 2026-09-28
dateModified: 2026-09-28
category: "Programming"
tags: ["Algorithms", "Python", "Computer Science", "Search"]
keywords: ["how do search engines work", "how does google search work", "what is an inverted index", "tf-idf explained", "what is pagerank", "build a search engine in python", "how search engines rank pages"]
readTime: "8 min read"
author: "Modern Age Coders Team"
---
# How Do Search Engines Work? Build a Tiny One in Python

> Crawling, indexing and ranking, explained by building a small search engine over this blog and running PageRank on its real internal links.

![How search engines work: a search box above a list of ranked results](/images/blog/how-do-search-engines-work/00-hero.png)

*By Modern Age Coders Team · 2026-09-28 · 8 min read*

**Quick answer:** Search engines crawl the web by following links, build an inverted index from every word to the pages that use it, rank matching pages when you search, and serve the best first. A classic ranking method is TF-IDF: pages score higher for words they use often, and rare words count more. Links matter too: PageRank treats links as votes weighted by the importance of the linking page. On this blog's real links, a post with 1 inbound link made the top five while another had 18, because link quality beats quantity. Modern engines add many more signals.

You type a few words, press Enter, and in well under a second a search engine picks the most useful pages out of billions. It feels like magic, but the core ideas are surprisingly approachable, and you can build a working miniature version in about forty lines of Python.

That is what this guide does. We build a small search engine over this very blog (it had 165 posts when we ran the code in September 2026), rank results with a classic formula called TF-IDF, and then run PageRank, the algorithm behind Google's early success, on the real links between our posts. The results, including the embarrassing ones, show exactly why ranking is the hard part.

## The four jobs of a search engine

![Four jobs of a search engine: crawl pages by following links, index which words appear where, rank matching pages for the query, and serve the best results first](/images/blog/how-do-search-engines-work/01-pipeline.png)

*The heavy work happens before you ever search.*

- **Crawling:** programs called crawlers or spiders download a page, find its links, and follow them to more pages, again and again.
- **Indexing:** each page's words are recorded in a giant lookup table, so the engine never has to read the pages again at search time.
- **Ranking:** when you search, every page that matches is given a score, and the scores decide the order.
- **Serving:** the top results are shown, usually with a title and a short description taken from the page.

## Step 1 and 2: crawl and index

Our "crawl" is simple: instead of downloading the web, we read the title and description of every post on this blog. Then we build an **inverted index**: for every word, a list of the posts that contain it and how often. It is called inverted because it flips the pages around: instead of page to words, it goes from word to pages, exactly like the index at the back of a book.

**search.py**

```python
import json, math, re, pathlib
from collections import defaultdict, Counter

# 1. "Crawl": read every blog post on this site (title and description)
docs = {}
for f in sorted(pathlib.Path("content/blog/data").glob("*.json")):
    meta = json.loads(f.read_text(encoding="utf-8"))["meta"]
    docs[meta["slug"]] = meta["title"] + " " + meta["description"]

STOP = {"a", "an", "and", "the", "of", "to", "in", "for", "on", "with", "is", "how", "what", "your", "you", "it", "from", "by", "or", "are", "at"}
def words(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

# 2. Index: for every word, which documents contain it, and how often
index = defaultdict(dict)
for slug, text in docs.items():
    for word, count in Counter(words(text)).items():
        index[word][slug] = count
print(f"indexed {len(docs)} posts, {len(index):,} distinct words")
print("posts containing 'fractions':", len(index["fractions"]))
print("posts containing 'python':   ", len(index["python"]))

# 3. Rank: TF-IDF, so rare words count for more than common ones
def search(query, top=3):
    scores = Counter()
    for word in words(query):
        postings = index.get(word, {})
        if not postings:
            continue
        idf = math.log(len(docs) / len(postings))
        for slug, tf in postings.items():
            scores[slug] += tf * idf
    return scores.most_common(top)

for w in words("fractions for children"):
    print(f"idf of {w!r}: log({len(docs)} / {len(index[w])}) = {math.log(len(docs) / len(index[w])):.2f}")

for q in ["fractions for children", "python projects beginners", "olympiad proof"]:
    print(f"\nsearch: {q!r}")
    for slug, score in search(q):
        print(f"  {score:5.2f}  {slug}")
```

**Output**

```text
indexed 165 posts, 1,278 distinct words
posts containing 'fractions': 1
posts containing 'python':    42
idf of 'fractions': log(165 / 1) = 5.11
idf of 'children': log(165 / 6) = 3.31

search: 'fractions for children'
  10.21  how-to-help-a-child-understand-fractions
   3.31  how-coding-improves-mathematical-thinking-children
   3.31  how-to-make-maths-fun-for-kids

search: 'python projects beginners'
   9.08  30-plus-scratch-project-ideas-kids-fun-coding-beginner-advanced
   9.04  python-for-beginners
   7.67  file-built-in-methods-python-guide

search: 'olympiad proof'
   8.83  pigeonhole-principle-explained
   8.01  how-to-write-a-maths-proof
   8.01  pythagoras-theorem-explained
```

![Inverted index of this blog: the word fractions points to 1 post and the word python points to 42 posts, out of 165 posts and 1,278 distinct words](/images/blog/how-do-search-engines-work/02-index.png)

*Finding every page with a word is a single lookup, not a search through every page.*

The index holds 1,278 distinct words across 165 posts. "fractions" appears in just 1 post, while "python" appears in 42. Common words such as "the" and "how" are dropped as *stop words*, because they appear everywhere and help with nothing.

## Step 3: rank with TF-IDF

Finding matching pages is easy. Putting them in the right order is hard. A classic starting point is **TF-IDF**: term frequency times inverse document frequency. A page scores higher when it uses a query word often (TF), and each word counts for more when it is rare across all pages (IDF). A word in every page tells you nothing; a word in one page tells you a lot.

![TF-IDF for the query fractions for children: fractions appears in 1 of 165 posts, idf 5.11; children appears in 6 posts, idf 3.31; the top result is how-to-help-a-child-understand-fractions with score 10.21](/images/blog/how-do-search-engines-work/03-tfidf.png)

*The rare word does most of the work.*

For "fractions for children", the top result was our guide to helping a child understand fractions, with a score of 10.21, well clear of the rest. "fractions" appears in only 1 post, so its IDF is 5.11, while "children" appears in 6 posts and scores 3.31. That is a good result.

## Where the tiny search engine goes wrong

Now look at "python projects beginners". The top result was `30-plus-scratch-project-ideas-kids-fun-coding-beginner-advanced`, a Scratch post, just ahead of our Python beginners guide. Why? Our engine matches exact words only. It does not know that "project" and "projects" are the same word, that Scratch is not Python, or what the searcher actually wants. It simply added up scores for the words it could match.

- **Word forms:** real engines treat "project", "projects" and "projecting" as related, a technique called stemming.
- **Meaning:** modern engines use language models to understand that "learn to code" and "programming for beginners" mean nearly the same thing.
- **Quality and trust:** a page that matches the words is not necessarily good. That is where links come in.

## PageRank: the idea that changed search

In 1998 Larry Page and Sergey Brin, then students at Stanford, described a way to measure a page's importance from the links pointing to it. The idea: a link is a vote, but votes from important pages count for more, and a page that links to many others spreads its vote thinly. Imagine a surfer clicking random links forever; PageRank is roughly the share of time they would spend on each page. We ran it on the real links between posts on this blog, as they stood in September 2026:

**pagerank.py**

```python
import json, re, pathlib

# Build the link graph: which blog posts link to which other blog posts
posts, links = {}, {}
for f in sorted(pathlib.Path("content/blog/data").glob("*.json")):
    data = json.loads(f.read_text(encoding="utf-8"))
    slug = data["meta"]["slug"]
    posts[slug] = data["meta"]["title"]
    text = json.dumps(data["content"]["sections"])
    links[slug] = set(re.findall(r"/blog/([a-z0-9-]+)", text))
for slug in links:
    links[slug] = {t for t in links[slug] if t in posts and t != slug}

# PageRank: a page is important if important pages link to it
n, d = len(posts), 0.85
rank = {s: 1 / n for s in posts}
for _ in range(50):
    new = {s: (1 - d) / n for s in posts}
    for s, outs in links.items():
        if outs:
            share = d * rank[s] / len(outs)
            for t in outs:
                new[t] += share
        else:                                  # a page with no links shares evenly
            for t in posts:
                new[t] += d * rank[s] / n
    rank = new

inbound = {s: sum(s in outs for outs in links.values()) for s in posts}
total_links = sum(len(o) for o in links.values())
print(f"{n} posts, {total_links:,} internal links between them")
print("top 5 by PageRank:")
for s in sorted(rank, key=rank.get, reverse=True)[:5]:
    print(f"  {rank[s] * 1000:5.2f}  {inbound[s]:>3} inbound  {s}")
```

**Output**

```text
165 posts, 385 internal links between them
top 5 by PageRank:
  51.77    6 inbound  usaco-bronze-to-silver-what-blocks-most-students
  48.31    5 inbound  big-o-notation-explained-simply
  33.72   18 inbound  python-for-beginners
  31.56    5 inbound  how-to-use-chatgpt-to-study-without-cheating
  28.09    1 inbound  how-to-teach-kids-ai-at-home
```

![Top five posts on this blog by PageRank across 385 internal links; how-to-teach-kids-ai-at-home ranks highly with only 1 inbound link, while python-for-beginners has 18](/images/blog/how-do-search-engines-work/04-pagerank.png)

*A single link from an important page can outweigh many ordinary ones.*

Look at the inbound counts. `python-for-beginners` has 18 posts linking to it, the most in the top five. Yet `how-to-teach-kids-ai-at-home` makes the top five with just 1. Its one link comes from a page that is itself highly ranked and links to very few other posts, so it passes on a large share of its importance. That is PageRank's key insight: the quality of links matters more than the quantity.

> **How modern search differs**

> Today's search engines use many signals beyond words and links, including how fresh a page is, whether it works well on a phone, and language models that interpret the meaning of a query. Google's own guide to its ranking systems says PageRank has evolved a lot since launch and remains part of its core ranking systems, alongside many others. The core pipeline of crawl, index and rank is still the same.

## Why this matters for you

- **Searching better:** knowing that rare, specific words carry the most weight helps you write sharper searches. "Fractions for 8 year olds" beats "help with maths".
- **Judging results:** the top result is the one a formula scored highest, not a guarantee of truth.
- **Learning to code:** a search engine uses dictionaries, loops, sorting and a little logarithm maths. It is a great project once you know the basics, and it connects naturally to [data structures and algorithms](/blog/how-to-start-learning-data-structures-and-algorithms).

> Finding the pages that match is easy. Deciding which one you actually wanted is the whole game.

## How we teach it

A search engine is a good example of learning by building, one of the principles on our [how we teach](/how-we-teach) page: forty lines of Python teach dictionaries, loops and sorting in a way no worksheet can. Tracing the code line by line until every step can be predicted is how a result like the Scratch post topping a Python search stops being a mystery. Our [Python course for teens](/courses/python-complete-masterclass-teens) runs one to one or in small groups of 5 to 10.

[Book a free class](/book-demo) [Book a priority demo](/book-demo)

## Frequently asked questions

**How do search engines work in simple terms?**

They do four jobs: crawl the web by following links, index which words appear on which pages, rank the matching pages when you search, and show the best ones first. Crawling and indexing happen in advance, so searching is fast.

**What is an inverted index?**

It is a lookup table from each word to the list of pages that contain it, like the index at the back of a book. It lets a search engine find every page with a word instantly instead of reading every page.

**What is TF-IDF?**

Term frequency times inverse document frequency. A page scores higher for a word it uses often, and each word counts for more if it is rare across all pages. It is a classic way to rank text results.

**What is PageRank?**

PageRank measures a page's importance from the links pointing to it. Links from important pages count for more, and a page with many outgoing links passes on less to each. It was described by Larry Page and Sergey Brin in 1998.

**Does Google still use PageRank?**

Yes, in an evolved form. Google's guide to its ranking systems says PageRank has changed a lot since Google launched and continues to be part of its core ranking systems, alongside many other systems.

**Why are some search results wrong?**

Ranking is a prediction of what you want, based on words, links and other signals. A page can match your words without answering your question, and a simple engine that matches exact words, like ours, can be fooled easily.

**Can I build my own search engine?**

Yes, a small one. With Python dictionaries you can build an inverted index and rank results with TF-IDF in about forty lines, as this post shows. It is an excellent intermediate coding project.

---

*Source: https://learn.modernagecoders.com/blog/how-do-search-engines-work/*