TopolearnMixing these up gives wrong validation numbers, random predictions or out-of-memory errors at inference.
Two independent switches control inference, and you usually want both.
`model.train()` / `model.eval()` set a flag on every sub-module. Only some layers react: Dropout stops dropping, and BatchNorm uses its running statistics instead of the batch's. Forgetting eval() makes predictions random (dropout) or batch-dependent (batch norm).
`torch.no_grad()` is a context manager (also a decorator) that stops autograd from recording operations. Outputs have requires_grad=False, no graph is kept and memory use drops. `torch.inference_mode()` is a stricter, slightly faster version: tensors created inside cannot be used in autograd later. Use it for pure inference and serving.
`t.detach()` returns a tensor sharing data with t but cut off from the graph. Use it when you want a value without its history: logging, storing outputs across steps, or feeding a target network.
A typical evaluation helper:
@torch.no_grad()
def evaluate(model, loader):
model.eval()
...Remember to switch back with model.train() before the next training epoch, since eval() persists until changed.
Example
import torch
from torch import nn
model = nn.Sequential(nn.Linear(100, 100), nn.Dropout(p=0.5))
x = torch.ones(1, 100)
model.train()
print(torch.equal(model(x), model(x))) # False: different dropout masks
model.eval()
print(torch.equal(model(x), model(x))) # True: dropout disabled
with torch.no_grad():
y = model(x)
print(y.requires_grad) # False
with torch.inference_mode():
z = model(x)
print(z.is_inference()) # True
w = model[0].weight * 2
print(w.requires_grad, w.detach().requires_grad) # True FalseRemember
eval()changes layer behaviour;no_grad()stops graph recording- Use both for validation and inference
detach()keeps the values but drops the history
Watch out for
- Accumulating
total_loss += lossacross steps, which keeps every graph alive; useloss.item()orloss.detach().