The Ramsgate projectFive ways to start a network, and what each one learned
1,326 land heights, one small network, and only the starting numbers changed.
The learner asks OpenTopoData for the height of the ground at 1,681 points, a 41 by 41 grid over a rectangle round Ramsgate, taken from the European Union's EU-DEM elevation model. 340 points come back with no height because they are out at sea, and 15 more are dropped at half a metre or less, leaving 1,326 points on land. Their average height is 40.72 m and the highest is 56.05 m. A random fifth, 266 points, is set aside for testing, and the network never sees them while training.
The network is small enough to write by hand in numpy: two inputs, the position of a point; two hidden layers of 32 units, each squashing its sum with tanh; one output, the predicted height. It trains for 3,000 steps with the Adam optimiser, the same data and the same settings every time. The only thing that changes is how its weights are filled at the start. Xavier initialisation, from Xavier Glorot and Yoshua Bengio in 2010, draws random weights scaled to the number of connections so that signals neither fade nor explode as they pass through the layers.
Height prediction error on the 266 test points (root mean square, metres), our numpy run; random schemes averaged over 5 seeds| Starting weights | Test error | Different first-layer units | What went wrong |
|---|
| Just predict the average height | 10.37 m | none | Baseline |
| Every weight 0 | 10.37 m | 1 of 32 | No gradient reaches the hidden units |
| Every weight 0.5 | 7.48 m | 1 of 32 | All 32 units stay identical |
| Random, very small (0.001) | 2.18 m | 32 of 32 | Slow start, rescued by Adam |
| Xavier random | 1.97 m | 32 of 32 | Works as intended |
| Random, very large (10) | 59.08 m | 32 of 32 | 65% of outputs stuck at the limit |
With every weight at zero, the network ends exactly where a person guessing the average would: 10.37 m out. The hidden units all output zero, so no correction ever flows back to them. With every weight at 0.5 it does a little better, 7.48 m, but its 32 first-layer units receive identical updates at every step and finish as 32 copies of one unit, so the network is really one neuron wide. Random starts break that symmetry. Xavier scaling gives the lowest error, 1.97 m on average, and a very small random start comes close at 2.18 m because the optimiser adapts its step sizes. Weights drawn with a spread about forty times the Xavier size push 65% of the tanh outputs against their limits, and the error, 59.08 m, is far worse than guessing.
Ages 8 to 11
Play a team guessing game where everyone copies the same answer, then one where each person starts differently.
Ages 11 to 15
Fit a straight line by nudging two numbers in Python, starting from different guesses.
Ages 15 and up
Write the network in numpy, try each start, and count the identical units yourself.
Data and credits
Heights are from EU-DEM v1.1, produced with funding by the European Union under the Copernicus programme, served by OpenTopoData and read on 1 October 2026. Xavier initialisation is from Glorot and Bengio, AISTATS 2010. The grid, the network, the training runs and every error figure above are our own, and a 25 metre elevation model is too coarse to show individual cliffs or streets.
