TopolearnMemory layout explains view errors, silent aliasing bugs and many performance differences.
A tensor is a view onto a flat storage: a pointer plus a storage_offset, a shape and a stride tuple. Element [i, j] lives at offset + i * stride[0] + j * stride[1].
Many operations only create a new view with different metadata and no copy: slicing, view, transpose, permute, expand, unsqueeze, narrow, diagonal. They share storage with the original, so writing through one changes the other.
A tensor is contiguous when its strides are the row-major ones for its shape. After a transpose it is not, and view (which never copies) may fail; reshape copies in that case, and .contiguous() makes a compact copy explicitly. expand uses stride 0 to repeat data without memory, which is why writing into an expanded tensor is an error.
Layout affects speed. Kernels prefer contiguous inputs, and convolutions on modern GPUs are fastest in channels_last layout: the logical shape stays N, C, H, W, but strides store channels innermost. Convert both model and inputs with .to(memory_format=torch.channels_last).
Check aliasing with untyped_storage().data_ptr(), and remember that small slices of huge tensors keep the whole storage alive; .clone() the slice if you want to free the rest.
Example
import torch
x = torch.arange(6).view(2, 3)
print(x.stride()) # (3, 1)
t = x.t() # transpose: new strides, same memory
print(t.stride(), t.is_contiguous()) # (1, 3) False
print(t.untyped_storage().data_ptr() == x.untyped_storage().data_ptr()) # True
try:
t.view(6)
except RuntimeError:
print("view() needs compatible strides")
print(t.contiguous().view(6)) # tensor([0, 3, 1, 4, 2, 5])
e = torch.tensor([1, 2, 3]).expand(4, 3)
print(e.stride()) # (0, 1) rows repeat without copying
img = torch.randn(8, 3, 32, 32).to(memory_format=torch.channels_last)
print(img.shape, img.stride()) # torch.Size([8, 3, 32, 32]) (3072, 1, 96, 3)Remember
- Tensor = storage + offset + shape + strides
- Views share memory;
contiguous()/clone()copy - channels_last keeps NCHW shape but stores C innermost
Watch out for
- Keeping a small slice of a huge tensor around, which keeps the entire storage in memory.