TopolearnUnderstanding graphs lets you debug slowness, build custom collections and reason about memory use.
Every Dask collection ultimately produces a task graph: a mapping from keys to tasks. Keys are strings or tuples like ("sum-chunk-abc123", 0), typically one per partition. A task is a function call whose arguments may reference other keys. In recent versions tasks are Task objects from dask._task_spec; in older code they were tuples like (func, arg1, arg2). You can still pass a plain dict graph to a scheduler: dask.get(graph, key).
Collections implement the Dask collection protocol: __dask_graph__(), __dask_keys__(), __dask_optimize__ and __dask_postcompute__. That's what lets dask.compute handle DataFrames, Arrays, Delayed objects and third-party collections like xarray together and merge their graphs.
Before running, the scheduler side does more work:
- Culling drops tasks not needed for the requested keys.
- Fusion merges linear chains of tasks into one to cut overhead.
- Ordering (
dask.order) picks an execution order that releases intermediate results early, which keeps memory low. Many memory blowups are ordering problems on unusual graph shapes.
Graph size is a cost in itself: building, sending and scheduling a graph of 10 million tasks takes noticeable time on the client and scheduler. Reduce task counts with bigger partitions, fused operations and fewer intermediate steps.
Example
import dask
from operator import add, mul
# A raw graph: keys map to values or tasks (func, *args); args can be keys
graph = {
"x": 1,
"y": 2,
"sum": (add, "x", "y"),
"out": (mul, "sum", 10),
"unused": (mul, "y", 100),
}
print(dask.get(graph, "out")) # 30 (synchronous scheduler)
@dask.delayed
def inc(i):
return i + 1
total = dask.delayed(sum)([inc(i) for i in range(3)])
g = dict(total.__dask_graph__())
print(len(g)) # 4 (three inc tasks + sum)
print(total.key.startswith("sum-")) # True
print(dask.compute(total)[0]) # 6Remember
- A graph maps keys to values or tasks that reference other keys
- Collections expose graphs via the
__dask_graph__protocol - Culling, fusion and ordering shape speed and memory
Watch out for
- Ignoring graph size: tens of millions of tasks can take minutes before any computation starts.