CS Map

Backpropagation

Also known as: backprop, reverse-mode autodiff, error backpropagation

intermediate concept 5 min read · Updated 2026-06-17

The algorithm that computes how much each weight in a neural network contributed to the error — applying the chain rule layer by layer in reverse so gradient descent knows which way to adjust every parameter.

Primary domain
Machine Learning
Sub-category
Supervised & Unsupervised Learning

In simple terms

Training a neural network requires knowing: “if I nudge weight W by a tiny amount, how much does the error change?” That question is answered by backpropagation — it sweeps through the network backwards from the output, computing the answer for every single weight in one efficient pass. Without it, you’d need a separate forward pass per weight to estimate each gradient numerically, making training of million-parameter networks impossible.

The Visual Map

graph LR
    x["x"] -->|forward| h["h = f(W₁·x)"] -->|forward| y["ŷ = g(W₂·h)"] --> L["Loss L"]
    L -.->|"∂L/∂ŷ"| gy["grad at output"]
    gy -.->|"× ∂ŷ/∂h (chain rule)"| gh["grad at hidden"]
    gh -.->|"× ∂h/∂W₁"| gw["∂L/∂W₁ for every weight"]
    cache["Forward caches activations\n→ reused in backward pass"] --- h

More detail

A neural network is a composition of functions: input → layer 1 → layer 2 → … → output → loss. Computing a loss given inputs is the forward pass; finding how the loss changes with respect to each weight — ∂L/∂W — is the backward pass, done by backpropagation. The mathematics is the chain rule applied to a computation graph: for L = f(g(h(x))), dL/dx = (dL/df)·(df/dg)·(dg/dh)·(dh/dx).

Backpropagation applies this starting from the loss and working backwards: compute the gradient of the loss with respect to the output activations; use the chain rule to propagate it back through each layer’s weights and activations; accumulate ∂L/∂W for every weight; then hand those gradients to gradient descent. The key efficiency insight is that each intermediate gradient is computed once and reused downstream — this is reverse-mode automatic differentiation, what PyTorch/TensorFlow/JAX implement as backward(). The forward pass caches intermediate values the backward pass needs, and the backward pass computes all n parameter gradients in O(n) work rather than O(n²). For transformers and other modern architectures the gradient flows through attention and residual connections — the same chain rule on a more complex graph. This is why deep learning is tractable: one forward plus one backward pass trains the whole network.

Under the Hood

Backprop is the chain rule, bookkept. This trains a tiny two-weight network (ŷ = w₂·relu(w₁·x)) on one example by computing each local derivative and multiplying them together — no framework, just the chain rule by hand:

def relu(z): return max(0.0, z)
def drelu(z): return 1.0 if z > 0 else 0.0

x, target = 2.0, 10.0
w1, w2, lr = 0.5, 0.5, 0.01

for step in range(2001):
    # forward (cache the intermediates)
    z = w1 * x
    h = relu(z)
    yhat = w2 * h
    loss = (yhat - target) ** 2

    # backward: multiply local derivatives along the chain
    dL_dyhat = 2 * (yhat - target)
    dL_dw2   = dL_dyhat * h                 # ∂ŷ/∂w2 = h
    dL_dh    = dL_dyhat * w2                # ∂ŷ/∂h  = w2
    dL_dw1   = dL_dh * drelu(z) * x         # through relu, then ∂z/∂w1 = x

    w2 -= lr * dL_dw2                        # gradient descent step
    w1 -= lr * dL_dw1
    if step % 500 == 0:
        print(f"step {step:>4}: ŷ={yhat:.3f} loss={loss:.4f}")

loss.backward() in PyTorch does exactly this multiplication of local derivatives, automatically, across millions of weights.

Engineering Trade-offs

  • Memory vs recompute. Backprop must cache forward activations to reuse them; that costs memory proportional to depth. Gradient checkpointing trades memory for recomputing some activations in the backward pass.
  • Reverse vs forward mode. Reverse mode is O(1) passes for many parameters and one output (the ML case); forward mode is cheaper when there are few inputs and many outputs — the opposite regime.
  • Numerical stability vs depth. Long chains multiply many derivatives; saturating activations make them vanish or explode — fixed by ReLU, normalisation, and residual connections, not by the algorithm.
  • Exactness vs cost. Backprop gives exact gradients cheaply; finite-difference checking is simple but costs a forward pass per parameter and is only used to validate.

Real-world examples

  • PyTorch’s loss.backward() runs backpropagation through the whole computation graph, populating .grad on every leaf tensor.
  • Training a 175-billion-parameter model is only feasible because backprop scales linearly with parameter count.
  • Gradient-based adversarial attacks (FGSM) backpropagate to the input instead of the weights, finding perturbations that maximise loss.

Common misconceptions

  • “Backprop is the same as gradient descent.” Backprop computes the gradients; gradient descent uses them. Paired but distinct.
  • “Backprop only works for feedforward networks.” It works for any differentiable computation graph — recurrent, attention, residual — because it operates on the graph, not the architecture.
  • “Deep networks vanish gradients because of backprop.” Vanishing gradients are an architecture issue (saturating activations, no shortcuts); ReLU, batch norm, and residual connections fix it.

Try it yourself

Train a two-weight network by hand-coded backprop and watch the chain rule drive the loss down (python3 only):

python3 - <<'EOF'
relu=lambda z:max(0.0,z); drelu=lambda z:1.0 if z>0 else 0.0
x,t=2.0,10.0; w1=w2=0.5; lr=0.01
for s in range(2001):
    z=w1*x; h=relu(z); yhat=w2*h; loss=(yhat-t)**2
    dy=2*(yhat-t)
    w2-=lr*dy*h
    w1-=lr*dy*w2*drelu(z)*x
    if s%500==0: print(f"step {s:>4}: yhat={yhat:.3f} loss={loss:.4f}")
EOF

Learn next

Neighborhood

A visual companion to the relationships above. Click any node to visit that topic.