Machine learning, explained with the Central line

The chart shows 109 periods of service operated (the share of the scheduled service that actually ran) for the Central line, from 2018-19 P1 to 2026-27 P6. The simplest possible model ignores all context and always predicts the same value c. Which c should we pick? Move the slider. Each vertical stroke is an error, called a residual.

The mean squared error (MSE) averages the squares of those residuals. The small chart below shows the MSE for every value of c: a parabola whose lowest point sits exactly on the average, 88.5%. An optimist who always predicts 100% does much worse.

The worst period in the series, 2020-21 P1 at 27.3%, drags the mean down more than the median, because its huge residual gets squared. That period coincides with the first COVID-19 lockdown in spring 2020, when the Tube ran a reduced service. If we measure the absolute error (MAE) instead, the median (90.0%) wins. The choice of error function changes the "best" answer.

This naive model is called the baseline. Any more complex model has to beat it, or it is useless.

We add one piece of information known in advance: last period's figure. Each dot pairs one period (vertical axis) with the period before it (horizontal axis). The model becomes a line ŷ = a·x + b. On this data, the best line has a slope of about 0.70: a bad period tends to be followed by another below-average one, but less extreme.

Tune a and b by hand to bring the MSE down. The map on the right shows the MSE for every pair (a, b): the cost function, a valley whose lowest point is marked with a star. Click anywhere on the map to move the model there.

Gradient descent does the job on its own. At each step it measures the slope of the valley at the current point and steps downhill. The step size is the learning rate η. Run it as is: the values are around 80, so the valley is a long, thin canyon and the descent crawls along it. Push η one notch higher and it diverges.

Now tick "Standardise x". Rescaling x to mean 0 and standard deviation 1 turns the canyon into a round bowl, and the descent reaches the bottom in a few steps, even with a large η. This is why we almost always normalise features.

Now x is time and y is the service operated of each period. To follow curves, we give the model powers of x: ŷ = w₀ + w₁x + w₂x² + … + w_d·x^d. It is still a linear model… in its parameters w. The degree d sets how flexible it is.

The model only sees a random sample of periods (solid dots); the others (hollow) are the test set. Raise the degree. At first, training and test errors fall together. Then the curve starts threading through every training dot: the training error keeps falling while the test error climbs back up. The model is learning the noise by heart. That is overfitting.

Click "New sample" at a high degree: the curve changes completely from one sample to the next. That is variance. At degree 1 it barely moves but misses every wave: that is bias. Drag "Training periods" up: with more data, the same high degree behaves much better.

Finally, tick "Forecast the last two years". The model trains on the past only and must extend its curve into the future. Even a curve that looks fine on the past shoots off the chart: polynomials are terrible forecasters.

From here on the data is simulated, so that each model's behaviour is easy to see. Two features: a train's delay at departure, and the share of late trains on the line. Each dot is a trip, blue if it arrived on time, orange if it arrived 5 or more minutes late.

Logistic regression computes a linear score z = w₀ + w₁x + w₂y and turns it into a probability with the sigmoid. The boundary where p = 0.5 is a straight line. Click "Watch it learn" to see gradient descent move the line into place.

The threshold decides from which probability we announce "late". Lower it and you miss fewer delays (recall goes up) but cry wolf more often (precision goes down). Watch the confusion matrix.

Now switch to "Circles" or "XOR": no straight line separates the classes and accuracy stalls. Tick "Add x², y², x·y": with these new features the boundary can bend. Crafting the right features by hand is called feature engineering.

This model looks for no formula at all. For a new trip, it finds the k closest trips in the training data and lets them vote. Hover over the chart: the lines connect the hovered point to the neighbours consulted.

With k = 1, every training point paints its own colour around itself: 100% training accuracy, islands everywhere, and a test score that suffers. As k grows, the boundary smooths out. k plays the same role as the polynomial degree: it sets the flexibility, only in reverse.

A tree asks one question after another, such as "delay at departure ≤ 6 min?". At each node it picks the question that best separates the classes, the one that makes the two resulting groups as pure as possible. Its boundaries are therefore made of horizontal and vertical segments.

The maximum depth sets the flexibility. At depth 1, a single question. At depth 10, the tree carves little boxes around every isolated point. The learned rules are printed under the chart.

Click "Resample" several times: the tree is rebuilt on a draw with replacement of the same data. The rules change a lot for a small change in the data. A deep tree is unstable.

A random forest trains dozens of trees, each on a different resample and with some randomness in the choice of features. Their mistakes differ, so averaging them cancels much of the error. Add trees: the boundary smooths out and the test error falls.

Boosting works the other way round. The trees are small and built one after another, each one correcting the errors left by the previous ones. It mostly reduces bias. This is the family of XGBoost, LightGBM and CatBoost, often the best choice on tabular data.

Compare: a forest of 1 tree is the unstable tree of chapter 6; with 100 trees it becomes stable. Boosting with many deep trees ends up learning the noise.

A neuron is a logistic regression: a weighted sum followed by an activation function. A network stacks layers of neurons. The diagram shows what each hidden neuron "sees": those of the first layer each draw a straight boundary, those of the next layer combine them into curved shapes.

The network therefore learns its own features, where chapter 4 made you invent them by hand. On the spiral, try 2 neurons, then 8: with too few neurons the shape is impossible to draw.

Training reuses the gradient descent of chapter 2. Computing the gradient layer by layer, from the output back to the input, is called backpropagation. In the diagram, orange links are positive weights and blue ones negative; their thickness follows their strength.