Reproducibility and seeding

Comparing experiments and debugging require runs you can repeat.

Randomness enters training through weight initialisation, data shuffling, augmentation, dropout and some GPU kernels. To repeat a run:

Seed all generators: Python's random, NumPy and PyTorch. torch.manual_seed seeds the CPU and all CUDA devices.
Give the DataLoader its own seeded torch.Generator via generator=. Each worker process derives its PyTorch seed from it; seed NumPy and random inside workers with a worker_init_fn if your dataset uses them.
Call torch.use_deterministic_algorithms(True) to make PyTorch raise an error whenever an operation has no deterministic implementation, and set torch.backends.cudnn.benchmark = False so cuDNN does not pick kernels by timing. Some CUDA matmul paths also need the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8.

Deterministic mode can be slower, so many teams only enable it when debugging.

Even with all of this, results are only guaranteed identical on the same hardware, driver and PyTorch version. Different GPUs, batch sizes or numbers of workers change floating-point summation order. For research claims, report the mean and spread over several seeds rather than chasing bit-exact runs.

Example

import random
import numpy as np
import torch
from torch.utils.data import DataLoader

def seed_everything(seed: int) -> None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)         # CPU and all CUDA devices

seed_everything(42)
a = torch.randn(3)
seed_everything(42)
b = torch.randn(3)
print(torch.equal(a, b))            # True

g = torch.Generator().manual_seed(0)
loader = DataLoader(list(range(10)), batch_size=5, shuffle=True, generator=g)
print([batch.tolist() for batch in loader])   # same order on every run

torch.use_deterministic_algorithms(True)      # error on nondeterministic ops
torch.backends.cudnn.benchmark = False

Remember

Seed random, NumPy and torch; give DataLoaders a seeded generator
use_deterministic_algorithms(True) surfaces nondeterministic ops
Bit-exactness holds only on identical hardware and versions

Watch out for

Seeding only torch while the dataset's augmentation uses NumPy's global RNG.

Check yourself

What does torch.use_deterministic_algorithms(True) do when an op has no deterministic implementation?

Official docs: pytorch.org/docs/stable/notes/randomness.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks