TopolearnGradients are what training follows; autograd computes them for any Python code you write.
Autograd is PyTorch's automatic differentiation engine. When a tensor has requires_grad=True, every operation on it is recorded in a graph of functions. Calling .backward() on a scalar result walks that graph backwards and stores d(result)/d(tensor) in each leaf tensor's .grad.
Key facts:
- Parameters of
nnlayers already haverequires_grad=True; input data does not. backward()needs a scalar (like a loss). For a non-scalar output, reduce it first with.sum()or.mean().- Gradients accumulate: a second
backward()adds to.gradrather than replacing it. That is why training loops calloptimizer.zero_grad()every step. - Each result tensor has a
grad_fnnaming the operation that created it, e.g.AddBackward0. - By default the graph is freed after
backward(); calling it twice on the same graph raises an error unless you passretain_graph=True.
The graph is built dynamically as your Python runs, so loops, if statements and function calls all work naturally. This define-by-run design is what makes PyTorch easy to debug: you can print tensors and step through with a debugger at any point.
Example
import torch
x = torch.tensor(3.0, requires_grad=True)
y = x ** 2 + 2 * x # y = 15
y.backward()
print(x.grad) # tensor(8.) dy/dx = 2x + 2
print(y.grad_fn) # <AddBackward0 object at ...>
w = torch.tensor([1.0, -2.0], requires_grad=True)
c = torch.tensor([3.0, 4.0])
(w * c).sum().backward()
print(w.grad) # tensor([3., 4.])
(w * c).sum().backward() # a new graph, gradients add up
print(w.grad) # tensor([6., 8.])
w.grad = None # reset, like optimizer.zero_grad()Remember
- Set
requires_grad=Trueon leaves; call.backward()on a scalar - Gradients accumulate in
.graduntil you clear them - The graph is rebuilt on every forward pass (define-by-run)
Watch out for
- Forgetting to zero gradients, so each step uses the sum of all previous gradients.