Chapter 01 · The neuron
One artificial neuron, the building unit.
An artificial neuron is, mechanically, a function. It takes several numbers in, produces one number out. That is the whole object. The function is a weighted sum followed by a non-linearity. Three lines of arithmetic.
Each input xi is multiplied by a learned weight wi. The products are summed. A learned bias b is added. The result is passed through an activation function — typically the ReLU, which simply returns the input if positive and zero otherwise. The output is one number.
y = f(w1x1 + w2x2 + w3x3 + b)
(1.1)
The weights and the bias are what the network learns. The inputs come from outside. The activation function is fixed. Everything fancy about modern neural networks reduces, at the bottom, to billions of copies of this small function with different learned weights and biases.
The honest part
A "neuron" is a deeply misleading name. There is no biology in here. It is a weighted sum followed by a max function. Frank Rosenblatt called it a neuron in 1957 and the name stuck. Try not to be misled by it.
Chapter 02 · Networks
Multilayer networks — neurons in formation.
A single neuron cannot do much. The leap is to arrange many neurons into layers, and stack the layers. Each layer's outputs become the next layer's inputs. Information flows from the input layer, through one or more hidden layers, to the output layer.
Each neuron in a hidden layer takes inputs from every neuron in the previous layer. This is why these are called fully connected or dense layers. With three inputs and a five-neuron hidden layer, you have 3×5 + 5 = 20 weights and biases to learn just at that layer. With a thousand inputs and a thousand-neuron hidden layer, you have a million.
Stack enough layers and the network can represent arbitrarily complex functions. This is the universal approximation theorem: a sufficiently large feedforward network can approximate any continuous function to any precision. It does not tell us how to find the right weights. That is the rest of this article.
Chapter 03 · Loss
The loss function — how we measure wrong.
Before the network can learn, we need a number that tells us how wrong it currently is. We call this the loss. The loss is small when the network's output matches the target and large when it does not. Learning is, mechanically, the process of reducing the loss.
For classification, the standard loss is cross-entropy. For regression, it is mean squared error. There are many others. They all share one property: they take the network's prediction and the true target and return a single non-negative number.
L = (ypred − ytrue)²
(3.1)
The loss is, geometrically, a function in a very high-dimensional space — one dimension per parameter of the network. With ten million parameters, the loss is a function in a ten-million-dimensional space. We cannot visualise this. We can, however, find a direction that reduces the loss at any given point. That direction is the gradient.
Chapter 04 · Gradient descent
Gradient descent, geometrically.
Imagine the loss as a landscape — hills, valleys, ridges, in millions of dimensions. The network's current parameters place you somewhere on this landscape, at some altitude (the current loss). Learning is, mechanically, the process of walking downhill. At every step, look at the slope of the ground beneath you and take a small step in the steepest-downhill direction.
wnew = wold − η · ∂L/∂w
(4.1)
The slope of the loss with respect to a single weight is its partial derivative: ∂L/∂w. We subtract a small multiple (η, the learning rate) of this slope from the weight. This nudges the weight slightly in the direction that reduces the loss. Apply this update across every weight in the network, and the network has, technically, learned a little.
Do this once for each training example. Or, for efficiency, do it for a small batch of examples at once — this is stochastic gradient descent. Repeat for billions of batches. After enough steps, the parameters settle into a configuration where the loss is low. That configuration is, by definition, the trained model.
The single most surprising fact in this field
Gradient descent works. Loss landscapes in millions of dimensions have, in principle, no reason to be navigable by simple downhill walking. They could be jagged and trap-filled. Empirically, they turn out to be remarkably smooth — full of broad valleys you can reach from many starting points. Why this is true is an open research question.
Chapter 05 · Backprop
Backpropagation, derived from one rule.
For gradient descent to work we need ∂L/∂w for every single weight. With ten million weights, this would seem to require ten million separate calculations. The trick that makes deep learning tractable is that we can compute all of them in one backward pass through the network using a single rule from high school: the chain rule.
The chain rule says: if L depends on y, and y depends on w, then ∂L/∂w = (∂L/∂y) × (∂y/∂w). Apply this recursively from the output backward through every layer, and you can compute the gradient of every parameter in time roughly equal to one forward pass.
∂L/∂wj = ∂L/∂aL · ∂aL/∂aL−1 · … · ∂aj+1/∂wj
(5.1)
The product of derivatives, traced backward through the network's layers, is what gives the algorithm its name. The forward pass computes the loss; the backward pass computes the gradient of every parameter. PyTorch and TensorFlow do this for you with one line — loss.backward() — but underneath, it is just the chain rule applied a few million times.
# In PyTorch — the entire training step
for x, y_true in dataloader:
y_pred = model(x) # forward pass
loss = criterion(y_pred, y_true)
loss.backward() # gradient of every parameter
optimizer.step() # nudge parameters downhill
optimizer.zero_grad()
Five lines. The same five lines train GPT-4 and your nano-GPT.
Chapter 06 · Depth
Why depth matters — the compositionality argument.
A two-layer network can technically approximate any function. So why are modern networks dozens or hundreds of layers deep? The answer is compositionality. Real-world data has hierarchical structure. An image is composed of objects, which are composed of parts, which are composed of edges, which are composed of pixels. Each layer of a deep network learns to detect features one step further up this hierarchy.
You can see this directly in trained image networks. The first layer learns edges. The second learns simple textures. The third learns shapes. The fourth learns parts of objects. The fifth learns whole objects. The same pattern shows up in language models — early layers track syntax, middle layers track semantics, later layers track high-level meaning and reasoning.
Depth is not magic. It is the architecture's way of saying: learning compositional structure is easier than learning everything at once.
Chapter 07 · CNN
Convolutional networks — seeing.
For images, a fully connected layer is hopelessly wasteful. A 224 × 224 image has 50,176 pixels; the first hidden layer alone would have hundreds of millions of weights. We need an architecture that respects the structure of images: spatial locality, translation invariance, repeated features.
The solution, due to Yann LeCun in the 1980s, is the convolutional layer. Instead of every neuron looking at every pixel, each neuron looks at a small local patch (say, 3 × 3 pixels). The same small set of weights — called a kernel — is slid across the entire image, producing a "feature map" of where that kernel's pattern appears.
A CNN typically stacks many such convolutional layers. The first layer's kernels learn to detect edges. The second layer's kernels learn to detect combinations of edges (corners, textures). The third learns combinations of textures (parts). And so on, until the deepest layers respond to entire objects. This is exactly the compositional argument of Chapter 6, made concrete.
CNNs dominated computer vision from AlexNet in 2012 through 2020, when transformers — adapted from language to vision via the Vision Transformer (ViT) — began to overtake them on the largest scales. Both architectures remain in active use depending on data size and compute budget.
Chapter 08 · RNN
RNNs and the long-range problem.
Sequences — text, speech, time series — pose a different challenge from images. They have order. They have variable length. A model needs to remember earlier parts of the sequence to make sense of later parts. The first solution was the recurrent neural network: a network whose hidden state at time t depends on its hidden state at time t−1.
This worked, modestly, for short sequences. For long ones it failed for two reasons. First, gradients vanish: when you backpropagate through many time steps, repeated multiplication by small numbers makes the gradient effectively zero by the time it reaches the early steps. Second, RNN computation is inherently sequential: you cannot compute step t before computing step t−1. This made training painfully slow on modern hardware.
The LSTM (long short-term memory) of Hochreiter and Schmidhuber, 1997, partially solved the vanishing-gradient problem with an explicit memory cell. LSTMs ruled natural language processing through about 2017. Then they were replaced.
Chapter 09 · Transformer
The transformer, and what changed.
The 2017 paper "Attention is All You Need" introduced an architecture with no recurrence and no convolutions. Just stacked layers of self-attention, in which every position in the sequence simultaneously looks at every other position and decides what to attend to. The attention mechanism is described in mechanical detail in our pillar page How AI Actually Works and implemented from scratch in Build Your Own GPT.
Two things changed for the transformer that did not change for any prior architecture. First — it parallelises perfectly across sequence positions, which means modern GPU hardware can be used at very high efficiency. Second — gradient flow is dramatically shorter, because every position can attend to every other in one step rather than propagating through many.
The result was the scaling era. Between 2017 and 2026, transformer models grew from millions of parameters to trillions. The same architecture. The same training objective. Wider, deeper, longer-context, more data. The capabilities at the other end stunned everyone, including the labs building them.
The whole field, on one line
Neural network = neuron → layer → network → trained by backpropagation → deepened to capture hierarchy → specialised by architecture (CNN for spatial, RNN/transformer for sequential). Every modern AI system you have heard of fits inside this sentence.
If you have made it through this article, you now have the conceptual map of deep learning. The next two natural directions: code it — our build your own GPT walkthrough is exactly that. And place yourself on the path — our AI engineer roadmap tells you what to do this month, this quarter, this year, until the engineer you want to be is the engineer in the mirror.