TopolearnThe loss defines what the model learns; picking the wrong one, or feeding it the wrong format, silently ruins training.
A loss function turns predictions and targets into one scalar to minimise. The common ones:
nn.CrossEntropyLossfor single-label classification. It takes raw logits of shape(N, C)and class indices of shape(N,)as int64. It applies log-softmax internally, so do not put a softmax at the end of your model. It also accepts class probabilities as targets and haslabel_smoothing=.nn.BCEWithLogitsLossfor binary or multi-label problems: logits and float targets of the same shape. It fuses the sigmoid for numerical stability.nn.MSELossandnn.L1Loss(ornn.SmoothL1Loss) for regression.
All losses average over the batch by default (reduction="mean"); use "sum" or "none" to get per-sample values, for example to apply your own weights. CrossEntropyLoss also has weight= for imbalanced classes and ignore_index= for padding tokens.
Every loss has a functional twin in torch.nn.functional: F.cross_entropy(logits, y) is the same as the module.
Sanity check: at initialisation, cross-entropy over C balanced classes should be close to ln(C), about 2.3 for 10 classes. A much larger starting loss hints at a bug in scaling or labels.
Example
import torch
from torch import nn
logits = torch.tensor([[2.0, 0.5, -1.0],
[0.1, 0.2, 3.0]]) # raw scores, no softmax
targets = torch.tensor([0, 2]) # class indices, int64
print(nn.CrossEntropyLoss()(logits, targets)) # tensor(0.1755)
per_sample = nn.CrossEntropyLoss(reduction="none")(logits, targets)
print(per_sample.shape) # torch.Size([2])
bce = nn.BCEWithLogitsLoss()
print(bce(torch.tensor([0.0]), torch.tensor([1.0]))) # tensor(0.6931) = ln 2
mse = nn.MSELoss()
print(mse(torch.tensor([1.0, 2.0]), torch.tensor([1.5, 2.0]))) # tensor(0.1250)Remember
- CrossEntropyLoss: logits in, int64 class indices as targets
- BCEWithLogitsLoss for binary and multi-label, float targets
- Initial cross-entropy should be about ln(number of classes)
Watch out for
- Applying softmax before CrossEntropyLoss, which double-normalises and slows learning.