Learn PyTorch

A tensor is an n-dimensional array with a dtype and a device, much like a NumPy array that can also live on a GPU.

Build tensors with zeros, ones, arange, linspace, full, eye and the random factories, and seed them for repeatability.

Choose numeric precision with dtype, move tensors between CPU and GPU with .to(), and know the type-promotion rules.

Slice, mask, reshape, permute, squeeze and combine tensors to get data into the shape a layer expects.

Element-wise ops, matrix multiplication with @, reductions along a dim, broadcasting and in-place ops with a trailing underscore.

Convert between tensors and NumPy arrays, knowing when memory is shared and when it is copied.

How a tensor maps to memory: storage, offset and strides, why views are free, when contiguous() copies, and channels_last.

Mark tensors with requires_grad, call backward() on a scalar loss, and read gradients from .grad.

Know what model.eval() changes, when to use torch.no_grad or torch.inference_mode, and what detach() does.

Define your own forward and backward with torch.autograd.Function, save tensors on ctx, and verify with gradcheck.

The graph of grad_fn nodes, leaf tensors, saved tensors and version counters, and higher-order gradients with create_graph.

Subclass nn.Module, create layers in __init__, describe the computation in forward, and call the model like a function.

Linear, Conv2d, pooling, Embedding, normalization, Dropout and activations, and how their input and output shapes relate.

Build an image classifier from conv, activation and pooling blocks, compute feature-map sizes, and use torchvision.

Turn token ids into vectors, pad batches of sequences, and run LSTMs with batch_first and packed sequences.

Start from pretrained torchvision weights, freeze the backbone, replace the head, then optionally fine-tune with a small learning rate.

Attach functions to modules and tensors to inspect or modify activations and gradients without changing model code.

Use F.scaled_dot_product_attention for fast fused attention, understand the math behind it, and the built-in transformer layers.

Use CrossEntropyLoss with raw logits and int64 class labels, BCEWithLogitsLoss for binary or multi-label targets, MSELoss for regression.

Use torch.optim (SGD, Adam, AdamW) to update parameters from their gradients with zero_grad, backward, step.

Put it together: iterate over batches, forward, loss, backward, step, then evaluate under torch.no_grad.

Wrap data in a Dataset with __len__ and __getitem__, then let DataLoader batch, shuffle and load it in parallel.

Save state_dicts of the model and optimizer with torch.save, and load them safely with torch.load(weights_only=True).

Change the learning rate during training with torch.optim.lr_scheduler: step decay, cosine, warmup and OneCycle.

Cap the gradient norm to stabilise training, and accumulate gradients over micro-batches to simulate a bigger batch.

Seed every random source, control DataLoader randomness, and opt into deterministic algorithms when you need identical runs.

Move the model and each batch to the accelerator, write device-agnostic code, and avoid hidden synchronisations.

Run matmuls in bfloat16 or float16 with torch.autocast, scale fp16 losses with GradScaler, and enable TF32.

Speed up models with one line: torch.compile captures Python into graphs and generates fused kernels, with caveats about graph breaks and recompiles.

Find where time goes with torch.profiler, label regions with record_function, and benchmark correctly with torch.utils.benchmark.

The caching allocator, allocated versus reserved memory, fragmentation, memory snapshots, and activation checkpointing to trade compute for memory.

Train on several GPUs or machines with DDP: one process per GPU, launched by torchrun, with a DistributedSampler and gradient all-reduce.

Shard parameters, gradients and optimizer state across GPUs with fully_shard (FSDP2) to train models too large for one device.

How torch calls are routed to ATen kernels by dispatch keys, and how to register custom operators with torch.library that work with autograd and torch.compile.

Capture a model as a whole-graph ExportedProgram with dynamic shapes, save it as .pt2, and feed AOTInductor, ONNX or ExecuTorch.