Everyday analogy: You drive to a shop. Your speed depends on how hard you press the accelerator. The accelerator position depends on traffic. To know how traffic affects your speed, you multiply: (speed per accelerator press) × (accelerator press per traffic condition). That is the chain rule - multiplying local rates of change along a chain.

Computation Graphs - Visualising the Maths

Modern frameworks like PyTorch build a computation graph during the forward pass. Every operation - add, multiply, ReLU - becomes a node. Backpropagation then walks this graph in reverse, applying the chain rule at each node to compute gradients.

Think of it like a river system. The loss is the ocean at the end. Backprop traces every tributary upstream to find how much each source (weight) contributed to the final flow.

A Tiny Worked Example

Suppose L = (w × x - y)² with w = 2, x = 3, y = 10.

1. Forward: w × x = 6, then 6 - 10 = -4, then (-4)² = 16. Loss = 16.
2. Backward: dL/d(diff) = 2 × (-4) = -8, then d(diff)/d(wx) = 1, so dL/d(wx) = -8.
3. Finally, d(wx)/dw = x = 3, so dL/dw = -8 × 3 = -24.

The gradient of −24 tells us: increasing w will decrease the loss rapidly. That is exactly the signal we need to improve.

Gradient Flow Through Layers

In a deep network, gradients must travel through many layers. Each layer multiplies the gradient by its local derivative. This creates two dangerous failure modes:

Vanishing Gradients

If local derivatives are small (e.g., the sigmoid function saturates near 0 or 1), repeated multiplication makes gradients shrink towards zero. Early layers barely learn - they receive almost no signal. This plagued early deep networks.

Exploding Gradients

If local derivatives are large, gradients grow exponentially. Weights receive enormous updates and the network becomes unstable, producing NaN values.

Modern solutions include:

• ReLU activation - derivative is 1 for positive inputs, avoiding shrinkage.
• Residual connections (skip connections) - give gradients a highway to bypass layers.
• Batch normalisation - keeps values in a healthy range.
• Gradient clipping - caps gradients to prevent explosions.

How Weights Actually Update

Once backprop computes every gradient, the optimiser (next lesson) updates each weight:

w_new = w_old - learning_rate × gradient

The learning rate controls the step size. Too large and you overshoot; too small and training takes forever. The gradient tells you the direction; the learning rate tells you how far to step.

Why Backprop Matters

Every time ChatGPT improves its next-word prediction, every time a self-driving car refines its steering, backpropagation is running underneath. It is the algorithm that makes learning from mistakes mathematically precise.

Without backprop, we would have no efficient way to train networks with millions - or billions - of parameters.

Key Takeaways

• Backpropagation computes gradients by walking the computation graph in reverse.
• The chain rule multiplies local derivatives along each path.
• Vanishing gradients slow learning; exploding gradients destabilise it.
• Modern tricks (ReLU, skip connections, gradient clipping) keep gradient flow healthy.
• Backprop + an optimiser = the learning engine of all modern deep learning.

📚 Further Reading

• Andrej Karpathy - nn-zero-to-hero (micrograd) - Build backprop from scratch in Python
• 3Blue1Brown - Backpropagation Calculus - Beautiful visual explanation of the chain rule in neural networks
• CS231n Backprop Notes - Stanford's concise reference on computation graphs and gradient flow