TopolearnDask Array scales NumPy code to arrays larger than memory, such as climate, imaging and simulation data.
A Dask Array is a grid of NumPy arrays (chunks) behind a NumPy-like API: da.ones, da.random.default_rng().random(...), da.from_array(numpy_or_h5py_or_zarr, chunks=...), da.from_zarr(path). Element-wise math, broadcasting, slicing, reductions with axis=, matmul, many linalg functions and NumPy ufuncs all work, lazily.
Chunks are the key design choice:
- Aim for chunks of roughly 100 MB (for example
(4000, 4000)float64 is 128 MB).chunks="auto"picks a size for you. - Align chunks with how you access data: for per-row time series, chunk so each row's data is in few chunks.
- Operations that combine many chunks, such as transposes and some reductions, cost communication.
x.rechunk((1000, -1)) changes chunking (-1 means the whole axis in one chunk); it's expensive, so do it once and deliberately. x.chunks and x.chunksize show the layout, and printing an array in Jupyter shows a helpful diagram.
For custom per-chunk logic, use x.map_blocks(func), or da.map_overlap when each chunk needs a border of neighbouring values, as in image filters and stencils. Store big results with x.to_zarr(path) rather than computing them into memory.
Example
import dask.array as da
import numpy as np
rng = da.random.default_rng(42)
x = rng.random((20_000, 5_000), chunks=(5_000, 5_000)) # ~800 MB, 4 chunks
print(x.numblocks, x.chunksize) # (4, 1) (5000, 5000)
print(round(x.nbytes / 1e6)) # 800
col_means = x.mean(axis=0) # lazy reduction
centered = x - col_means # broadcasting, still lazy
print(np.allclose(centered.mean(axis=0).compute(), 0, atol=1e-9)) # True
y = x.rechunk((1_000, -1)) # whole rows per chunk
print(y.numblocks) # (20, 1)
z = x[:4, :3].map_blocks(np.round, 2)
print(z.compute().shape) # (4, 3)Remember
- A Dask Array is a grid of NumPy chunks with the NumPy API
- Chunks of around 100 MB aligned with your access pattern work best
rechunkis expensive;map_blocksandmap_overlaprun custom code per chunk
Watch out for
- Creating millions of tiny chunks (e.g. chunks=(10, 10)) and drowning in scheduler overhead.