Loss Functions and Optimisers
Loss Functions and Optimisers
Backpropagation gives us gradients - but gradients of what? Before backprop can run, we need a single number that captures how wrong the model is. That number comes from a loss function. Once we have gradients, an optimiser decides how to update the weights. Together, they form the learning loop.
What Is a Loss Function?
A loss function (also called a cost function) takes the model's prediction and the true answer, and returns a number measuring "wrongness." The goal of training is to minimise this number.
Think of it like a score in golf - lower is better. A loss of 0 means a perfect prediction.
Loss Functions for Regression - MSE
When predicting continuous values (house prices, temperature), we use Mean Squared Error (MSE):
MSE = (1/n) ร ฮฃ(predicted - actual)ยฒ
Squaring does two things: it makes all errors positive, and it punishes large errors disproportionately. Predict a house price off by ยฃ100k and the squared error is 100ร worse than being off by ยฃ10k.
MSE dates back to Carl Friedrich Gauss in 1795 - over two centuries before neural networks. He used it to track the orbit of the asteroid Ceres.
Loss Functions for Classification - Cross-Entropy
When predicting categories (spam or not, cat vs dog), we use cross-entropy loss. It measures how far the model's predicted probabilities are from the true labels.
If the correct answer is "cat" and the model says 99% cat, the loss is tiny. If it says 10% cat, the loss is enormous. Cross-entropy has a useful property: it becomes infinitely unhappy when the model is confidently wrong, creating a strong gradient to correct the mistake.
Binary cross-entropy is for two-class problems. Categorical cross-entropy handles multiple classes by comparing probability distributions.
Why is MSE a poor choice for classification tasks?
Discussion
Sign in to join the discussion