def build_a_gpt(): # in 300 lines of Python.

What we are building, exactly.

A character-level GPT. About 10 million parameters. Trained on a small text file — Shakespeare is the traditional choice. After roughly thirty minutes of training on a free Colab GPU, this model produces output that looks like Shakespeare. Not Shakespeare. Like Shakespeare. Pentameter-ish phrasing, mostly real words, occasional comedy.

This is enough scale to actually feel the magic. Big enough to learn meaningful patterns. Small enough to train in one sitting. Architecturally identical to GPT-2 — same transformer block, same multi-head attention, same training objective. The thing we will build is a real LLM. Tiny, but real.

Three files. Three concepts. One file for the model, one for training, one for sampling. The full file at the end of this article fits in your editor without scrolling.

The training data.

One text file. Plain text. The complete works of Shakespeare is the canonical choice — about 1 MB of clean English, broken into character-level structure. You can substitute any large text corpus you like. We have students train on Tagore, on Sherlock Holmes, on years of their own WhatsApp logs (with permission).

# Download the tiny Shakespeare dataset
import urllib.request

url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
urllib.request.urlretrieve(url, "input.txt")

with open("input.txt", "r") as f:
    text = f.read()

print(f"Total characters: {len(text):,}")
print(f"First 200 chars:")
print(text[:200])

1.1 million characters. About a quarter of a million tokens once we run our (tiny) tokenizer. Real Elizabethan English with stage directions, character names, line breaks. The model will learn the rhythm of dialogue, the rhythm of pentameter, the surface texture of seventeenth-century vocabulary.

A character tokenizer in twelve lines.

Real LLMs use byte-pair encoding (BPE) tokenizers with vocabularies of fifty thousand or more. We will use something simpler: one character = one token. The vocabulary becomes the set of unique characters in our training text. Tiny — about 65 entries. It is conceptually identical to a real tokenizer, just simpler.

# Build the vocabulary
chars = sorted(list(set(text)))
vocab_size = len(chars)
print(f"Vocab size: {vocab_size}")

# Map characters to integers and back
stoi = { ch: i for i, ch in enumerate(chars) }
itos = { i: ch for i, ch in enumerate(chars) }

def encode(s):
    return [stoi[c] for c in s]

def decode(l):
    return ''.join([itos[i] for i in l])

print(encode("hello"))
print(decode(encode("hello")))

Two dictionaries — one mapping character to ID, one mapping ID back to character. Encode and decode functions that walk through a string. That is the whole tokenizer. Twelve lines. Conceptually identical to what GPT-4's BPE tokenizer does, except ours has 65 entries and theirs has 100,257.

Token and position embeddings.

The model cannot look at a number 46 and know it means the letter "h". The first step inside the model is to convert each token ID into an embedding — a learnable vector of size n_embd, typically 384 in our tiny model. We need a second embedding too: a position embedding, which tells the model where each token sits in the sequence. Without position embeddings, the model treats sentences as bags of words.

import torch
import torch.nn as nn

n_embd = 384
block_size = 256  # max context length in tokens

# Token embedding: each token ID gets a vector of size n_embd
token_embedding_table = nn.Embedding(vocab_size, n_embd)

# Position embedding: each position 0..block_size-1 gets a vector
position_embedding_table = nn.Embedding(block_size, n_embd)

# Inside forward pass:
tok_emb = token_embedding_table(idx)        # (B, T, n_embd)
pos_emb = position_embedding_table(torch.arange(T))  # (T, n_embd)
x = tok_emb + pos_emb                       # (B, T, n_embd)

Each nn.Embedding is conceptually a lookup table — a matrix where row i is the embedding for ID i. The rows start random and are learned during training. By the end, the row for the letter "h" sits close, in embedding space, to the rows for "g", "i", "j" and other letters with similar contexts.

The addition tok_emb + pos_emb is one of the strangest and most beautiful tricks in the transformer. Position and content live in the same vector space, added together. The model learns to disentangle them in later layers. It works astonishingly well.

Single-head attention, derived from scratch.

This is the heart. Once you understand a single attention head, multi-head is just running several in parallel, and the rest of the architecture is plumbing.

The intuition first

For each position in the sequence, we want to look at previous positions and decide how much to attend to each. We project each token into three vectors: a query, a key, and a value. The query of the current token is dot-producted with the keys of all previous tokens to produce attention scores. The scores are softmaxed to become weights. The weighted sum of the values is the output.

class Head(nn.Module):
    def __init__(self, head_size):
        super().__init__()
        self.key   = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        self.register_buffer('tril',
            torch.tril(torch.ones(block_size, block_size)))

    def forward(self, x):
        B, T, C = x.shape
        k = self.key(x)    # (B, T, head_size)
        q = self.query(x)  # (B, T, head_size)
        v = self.value(x)  # (B, T, head_size)

        # Compute attention scores
        wei = q @ k.transpose(-2, -1) * C**-0.5  # (B, T, T)
        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
        wei = torch.softmax(wei, dim=-1)            # (B, T, T)

        out = wei @ v   # (B, T, head_size)
        return out

Three lines of math, doing all the real work:

• q @ k.transpose(-2, -1) — dot product of every query with every key. Big when they "align" in embedding space.
• * C**-0.5 — scaling, so softmax doesn't saturate at large head sizes.
• masked_fill(tril==0, -inf) — the causal mask. Each position can only attend to itself and earlier positions. This is what makes the model autoregressive.
• softmax — normalises scores into weights that sum to 1.
• wei @ v — the weighted sum.

If you understand these five lines, you understand attention. The rest of the paper is decoration. Read them twice.

Multi-head attention is just several heads, concatenated.

One head is good. Six heads is better. Each head learns to look at the sequence in a different way. One head learns about subject–verb agreement. Another tracks long-range dependencies. Another tracks rhyme. We don't tell them what to learn. They discover their specialisations during training.

class MultiHeadAttention(nn.Module):
    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList(
            [Head(head_size) for _ in range(num_heads)]
        )
        self.proj = nn.Linear(n_embd, n_embd)

    def forward(self, x):
        # Concatenate outputs of every head along the channel dim
        out = torch.cat([h(x) for h in self.heads], dim=-1)
        out = self.proj(out)
        return out

If n_embd=384 and num_heads=6, each head produces a tensor of size head_size=64. Concatenating six of them gives back 384. Then a final linear projection mixes them. That is multi-head attention. Six lines.

The transformer block.

A transformer block is attention followed by a small feedforward network, each wrapped in a layer-norm and a residual connection. This is the unit we will stack repeatedly to make the full model.

class FeedForward(nn.Module):
    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_embd, 4 * n_embd),
            nn.ReLU(),
            nn.Linear(4 * n_embd, n_embd),
        )
    def forward(self, x):
        return self.net(x)


class Block(nn.Module):
    def __init__(self, n_embd, n_heads):
        super().__init__()
        self.sa = MultiHeadAttention(n_heads, n_embd // n_heads)
        self.ffwd = FeedForward(n_embd)
        self.ln1 = nn.LayerNorm(n_embd)
        self.ln2 = nn.LayerNorm(n_embd)

    def forward(self, x):
        x = x + self.sa(self.ln1(x))     # residual
        x = x + self.ffwd(self.ln2(x))   # residual
        return x

That's it. The same block, repeated six times, is the entire body of our model. The feedforward layer (size 4× n_embd in the hidden) is where most of the model's "knowledge" actually lives. Attention moves information between tokens; the feedforward layer transforms that information per-token.

The full model class.

Stack the embeddings, the transformer blocks, a final layer norm, and a head that projects back to vocabulary size. The full GPT model in one class:

n_layer = 6
n_heads = 6

class GPT(nn.Module):
    def __init__(self):
        super().__init__()
        self.token_embedding = nn.Embedding(vocab_size, n_embd)
        self.pos_embedding   = nn.Embedding(block_size, n_embd)
        self.blocks = nn.Sequential(*[
            Block(n_embd, n_heads) for _ in range(n_layer)
        ])
        self.ln_f = nn.LayerNorm(n_embd)
        self.head = nn.Linear(n_embd, vocab_size)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        tok_emb = self.token_embedding(idx)
        pos_emb = self.pos_embedding(torch.arange(T))
        x = tok_emb + pos_emb
        x = self.blocks(x)
        x = self.ln_f(x)
        logits = self.head(x)   # (B, T, vocab_size)

        loss = None
        if targets is not None:
            B, T, C = logits.shape
            loss = nn.functional.cross_entropy(
                logits.view(B*T, C), targets.view(B*T)
            )
        return logits, loss

Roughly 10 million parameters. The full architecture of GPT-2 at small scale. Save it. Print sum(p.numel() for p in model.parameters()) to convince yourself the parameter count is in the right ballpark.

The training loop.

For each batch, we draw random chunks of block_size tokens from the training text. The targets are the same chunks shifted by one — the model's job is to predict the next token at every position. We compute cross-entropy loss across the whole sequence, backpropagate, and step the optimizer.

batch_size = 64
lr = 3e-4
max_iters = 5000
device = "cuda" if torch.cuda.is_available() else "cpu"

data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9*len(data))
train_data, val_data = data[:n], data[n:]

def get_batch(split):
    src = train_data if split=='train' else val_data
    ix = torch.randint(len(src) - block_size, (batch_size,))
    x = torch.stack([src[i:i+block_size] for i in ix])
    y = torch.stack([src[i+1:i+block_size+1] for i in ix])
    return x.to(device), y.to(device)

model = GPT().to(device)
optim = torch.optim.AdamW(model.parameters(), lr=lr)

for step in range(max_iters):
    xb, yb = get_batch('train')
    logits, loss = model(xb, yb)
    optim.zero_grad(set_to_none=True)
    loss.backward()
    optim.step()

    if step % 500 == 0:
        print(f"step {step:5d} | loss {loss.item():.4f}")

Loss falls from 4.2 (essentially uniform random over 65 chars, log(65) ≈ 4.17) toward 1.4 — meaningful prediction. Thirty minutes on a free Colab T4 GPU. Half a million parameter updates. The model is, slowly, learning Shakespeare.

Sampling — generating text.

The model takes a sequence and produces a probability over the next token. To generate, we sample one token from that probability distribution, append it to the input, and repeat. Greedy sampling (always pick the highest-probability token) produces dull, repetitive output. Sampling with a small temperature introduces controlled randomness and produces more interesting text.

@torch.no_grad()
def generate(model, idx, max_new_tokens, temperature=1.0):
    for _ in range(max_new_tokens):
        idx_cond = idx[:, -block_size:]
        logits, _ = model(idx_cond)
        logits = logits[:, -1, :] / temperature   # last position
        probs = torch.softmax(logits, dim=-1)
        next_id = torch.multinomial(probs, num_samples=1)
        idx = torch.cat([idx, next_id], dim=1)
    return idx

start = torch.zeros((1, 1), dtype=torch.long, device=device)
out = generate(model, start, max_new_tokens=300)
print(decode(out[0].tolist()))

Not Shakespeare. But — recognisably Shakespeare-shaped. Capital-letter character names. Colons. Dialog rhythm. Five-syllable-ish lines. Made-up words that sound plausible. This came out of 10 million parameters trained on 1 MB of text for thirty minutes. Now think about what GPT-4 — 1.7 trillion parameters trained on perhaps 10 trillion tokens — is doing. It is exactly the same algorithm. Scaled.

What scaling up actually means.

You have just built nano-GPT. The path from nano-GPT to GPT-2 is increasing five numbers: n_layer, n_heads, n_embd, block_size, and the training corpus size. GPT-2 had 12 layers, 12 heads, 768 embedding size, 1024 context. GPT-3 was the same architecture with vastly bigger numbers — 96 layers, 96 heads, 12288 embedding size — and trained on hundreds of billions of tokens for months on thousands of GPUs.

The path from GPT-3 to GPT-4 added: a real BPE tokenizer (50,000+ tokens), grouped-query attention, mixture-of-experts in some variants, rotary position embeddings (rather than learned ones), much more training data, and the RLHF stage on top. The skeleton you just wrote is still in there.

The full file, all 300 lines, copy-paste runnable.

Available on our GitHub at github.com/modernagecoders/nano-gpt-tutorial. One file. Three commands to run: download data, train, sample. We tested it on a fresh Colab notebook on 14 May 2026. It runs.

If you got this far — you have just built an LLM from scratch. Not a toy. Not a hello world. A real, working, autoregressive transformer language model. The same family of math behind ChatGPT. You will never read about "AI" in the news the same way again.

What's next? Two natural directions. One — keep going on language: switch to a real BPE tokenizer, train on a bigger corpus, scale up the layers. Two — take what you have learned and apply it to images, audio, agents, or fine-tuning. Both paths are inside our advanced courses below.