Backpropagation demystified: hand-compute the gradients, then verify with autograd

Why backpropagation still matters

Backpropagation is the engine behind modern deep learning. Whether you’re training a tiny multilayer perceptron or a frontier-scale language model, every optimization step depends on efficiently computing gradients.

The good news is that backpropagation is not magic. It is simply the chain rule applied systematically across a computational graph.

In this guide, you’ll:

• Build intuition for backpropagation.

• Hand-compute gradients through a tiny two-layer network.

• Verify every result with PyTorch Autograd.

• Learn how reverse-mode automatic differentiation works.

• See why gradient checking remains one of the most valuable debugging tools in machine learning.

By the end, you’ll understand not just what Autograd computes, but why the numbers are correct.

---

The core idea: the chain rule

Suppose a computation looks like this:

x
│
▼
a
│
▼
b
│
▼
loss

If the loss depends on b, which depends on a, which depends on x, then:

d(loss)/d(x) =
d(loss)/d(b)
× d(b)/d(a)
× d(a)/d(x)

Every derivative is a local derivative.

Backpropagation simply multiplies these local derivatives together while moving backward through the computation graph.

Instead of recomputing derivatives independently for every parameter, reverse-mode automatic differentiation reuses intermediate results. That’s why models with millions—or even billions—of parameters can be trained efficiently.

---

Computational graphs: what Autograd actually records

Every differentiable operation becomes a node in a computational graph.

Our tiny network looks like this:

Input x
   │
Linear (w1, b1)
   │
  z1
   │
 ReLU
   │
  a1
   │
Linear (w2, b2)
   │
Prediction ŷ
   │
Squared Error
   │
 Loss

During the forward pass, every node computes a value.

During the backward pass, every node receives a gradient from the node above and computes gradients for its own inputs.

Think of every edge as carrying two things:

• A value during the forward pass.

• A gradient during the backward pass.

That repeated application of the chain rule is the entire essence of backpropagation.

---

Our tiny neural network

We’ll deliberately keep the network small enough to compute every gradient by hand.

Architecture:

Input
  ↓
Linear
  ↓
ReLU
  ↓
Linear
  ↓
Prediction
  ↓
Squared Error Loss

Parameters:

• x = 2

• target = 1

• w1 = 3

• b1 = 1

• w2 = 4

• b2 = −2

---

Step 1: Forward pass

Hidden pre-activation:

z1 = w1 × x + b1
   = 3 × 2 + 1
   = 7

Apply ReLU:

a1 = max(0, z1)
   = 7

Output:

ŷ = w2 × a1 + b2
  = 4 × 7 − 2
  = 26

We’ll use the squared-error loss with the common 0.5 scaling:

Loss = 0.5 × (ŷ − target)²

Substituting the numbers:

Loss = 0.5 × (26 − 1)²
     = 0.5 × 25²
     = 312.5

Why multiply by 0.5?

Without the scaling:

Loss = (ŷ − target)²

the derivative becomes:

2 × (ŷ − target)

Multiplying by 0.5 cancels the extra factor of 2:

dLoss/dŷ = ŷ − target

The optimization behaves identically—the gradients are simply scaled consistently.

---

Step 2: Start from the loss

Always begin at the end.

Loss = 0.5 × (ŷ − target)²

Differentiate:

dLoss/dŷ = ŷ − target = 25

This is the first gradient flowing backward.

---

Step 3: Gradients of the second layer

Our output equation is:

ŷ = w2 × a1 + b2

For the weight:

dŷ/dw2 = a1 = 7

Therefore:

dLoss/dw2 = 25 × 7 = 175

For the bias:

dŷ/db2 = 1

So:

dLoss/db2 = 25

Now propagate the gradient toward the hidden layer:

dŷ/da1 = w2 = 4

Therefore:

dLoss/da1 = 25 × 4 = 100

---

Step 4: Through the ReLU

ReLU is defined as:

ReLU(x) = max(0, x)

Its derivative is:

• 1 when the input is positive

• 0 when the input is negative

Since:

z1 = 7

the derivative is:

dReLU/dz1 = 1

Therefore:

dLoss/dz1 = 100 × 1 = 100

ReLU acts like a gradient gate:

• Positive activations pass gradients through unchanged.

• Negative activations block gradients entirely.

This explains why “dead ReLU” neurons can stop learning if they never activate.

---

Step 5: First-layer gradients

Recall:

z1 = w1 × x + b1

For the weight:

dz1/dw1 = x = 2

Therefore:

dLoss/dw1 = 100 × 2 = 200

For the bias:

dz1/db1 = 1

Therefore:

dLoss/db1 = 100

Finally, the gradient with respect to the input:

dz1/dx = w1 = 3

Therefore:

dLoss/dx = 100 × 3 = 300

Gradients with respect to inputs are useful for:

• Saliency maps

• Feature attribution

• Adversarial examples

• Input optimization

---

Final gradient summary

Quantity	Gradient
w1	200
b1	100
w2	175
b2	25
x	300

Every one of these values should match what Autograd computes.

---

Verify everything with PyTorch Autograd

The following example works with the current stable PyTorch release and reproduces the manual calculations exactly.

import torch

# Inputs
x = torch.tensor(2.0, requires_grad=True)
target = torch.tensor(1.0)

# Parameters
w1 = torch.tensor(3.0, requires_grad=True)
b1 = torch.tensor(1.0, requires_grad=True)

w2 = torch.tensor(4.0, requires_grad=True)
b2 = torch.tensor(-2.0, requires_grad=True)

# Forward pass
z1 = w1 * x + b1
a1 = torch.relu(z1)
y_hat = w2 * a1 + b2

loss = 0.5 * (y_hat - target) ** 2

# Backward pass
loss.backward()

print(f"Loss = {loss.item():.1f}")
print(f"dw1 = {w1.grad.item()}")
print(f"db1 = {b1.grad.item()}")
print(f"dw2 = {w2.grad.item()}")
print(f"db2 = {b2.grad.item()}")
print(f"dx  = {x.grad.item()}")

Expected output:

Loss = 312.5
dw1 = 200.0
db1 = 100.0
dw2 = 175.0
db2 = 25.0
dx  = 300.0

Every gradient exactly matches the hand calculation.

---

What happens when you call backward()?

Calling:

loss.backward()

starts at the loss with an initial gradient of 1.

Autograd then traverses the graph in reverse topological order.

Each operation already knows its own local derivative:

• Addition contributes a derivative of 1.

• Multiplication remembers both operands.

• ReLU remembers which inputs were positive.

• Matrix multiplication computes gradients with respect to both inputs.

Autograd combines these local derivatives using the chain rule and accumulates the results into each tensor’s .grad field.

The forward pass builds the graph. The backward pass walks it in reverse.

---

Why gradients accumulate

One subtle behavior surprises almost everyone at first.

Gradients accumulate by default.

loss.backward()
loss.backward()

After the second call, every gradient doubles.

This is intentional. During training, gradients from multiple losses or mini-batches can be accumulated before updating the model.

A typical training loop clears gradients first:

optimizer.zero_grad(set_to_none=True)

loss = model(inputs)
loss.backward()
optimizer.step()

Using set_to_none=True is still the recommended approach because it avoids unnecessary memory writes and allows PyTorch to allocate gradient buffers lazily.

---

A practical debugging workflow

When a model refuses to learn, gradients are often the first thing to inspect.

A reliable workflow is:

1. Confirm every trainable parameter has requires_grad=True.

2. Verify that no tensor was accidentally detached from the graph.

3. Print gradient norms after backward().

4. Watch for exploding or vanishing gradients.

5. Reduce the model to the smallest reproducible example.

6. Run a numerical gradient check if you’ve written a custom operation.

Many difficult training bugs become obvious once you inspect where gradients stop flowing.

---

Common mistakes

Forgetting the chain rule

The derivative of one operation is almost never the final answer.

Every upstream gradient must be multiplied through the computation graph.

Ignoring activation functions

Activations determine how gradients flow.

For example:

• ReLU blocks negative activations.

• Sigmoid saturates for very large positive or negative values.

• Tanh can also produce very small gradients in saturated regions.

Activation choice often has a larger effect on optimization than newcomers expect.

Mixing forward and backward reasoning

The forward pass computes predictions.

The backward pass computes sensitivities.

Keeping those mental models separate makes debugging much easier.

Assuming larger gradients are always better

Very small gradients slow learning.

Very large gradients can make optimization unstable.

Healthy training usually means gradients stay within a reasonable range throughout the network.

---

Cherry on the cake: gradient checking has caught real framework bugs

It might seem like numerical gradient checking is only for students learning calculus—but major deep learning frameworks still rely on it during development.

PyTorch’s own test suite routinely uses torch.autograd.gradcheck, which compares analytical gradients against finite-difference approximations before new differentiable operators are accepted. Over the years, contributors have uncovered subtle bugs in custom C++ and CUDA kernels where the forward computation looked correct, but the backward pass silently produced incorrect gradients. These issues often escaped ordinary unit tests because the outputs were correct while only the derivatives were wrong.

The surprising part is that this technique continues to catch regressions even in a mature framework. Gradient correctness is difficult enough that every new differentiable operator is expected to prove its backward implementation numerically before it can be trusted.

The lesson is simple: if you’re writing a custom autograd.Function, a C++ extension, or a Triton kernel, make gradient checking part of your test suite from day one.

---

Key takeaways

• Backpropagation is the chain rule applied across a computation graph.

• Reverse-mode automatic differentiation makes gradient computation efficient for models with many parameters.

• Every operation contributes a local derivative during the backward pass.

• ReLU either passes or blocks gradients depending on its input.

• The 0.5 factor in squared-error loss simplifies the derivative without changing the optimization objective.

• PyTorch Autograd reproduces exactly the same gradients you can compute by hand.

• Numerical gradient checking remains one of the best ways to validate custom differentiable code.

---

Where to go next

A great follow-up exercise is to extend this example:

• Add a second hidden neuron.

• Replace ReLU with GELU or Tanh.

• Compute the gradients by hand first.

• Verify every result with Autograd.

• Finally, try torch.autograd.gradcheck on your own custom function to see numerical verification in action.

The fastest way to build intuition is to alternate between pencil-and-paper derivations and runnable code. Try the exercise yourself, compare your manual gradients with Autograd, and keep experimenting—once the chain rule clicks, you’ll understand the foundation of every modern neural network.