TopolearnMany real workloads aren't DataFrames or arrays, and delayed parallelizes them with a few decorators.
dask.delayed turns a function call into a lazy task. Calling a delayed function doesn't run it; it returns a Delayed object representing the future result. Passing Delayed objects into other delayed calls links them into a graph.
@dask.delayed
def load(path): ...
parts = [load(p) for p in paths]
total = dask.delayed(sum)(parts)
total.compute()Dask sees that all load calls are independent and runs them in parallel, then runs sum when they're done. You write a normal-looking loop; Dask extracts the parallelism.
Guidelines:
- Decorate the function, then call it; don't call the function and wrap the result (
delayed(f(x))runsfimmediately). - Each task should do a meaningful amount of work, at least around 100 ms. Each task costs scheduler overhead, so a million tiny tasks is slower than a plain loop.
- Don't mutate inputs or rely on global state inside tasks; they may run in another process or machine.
- Collect everything with one
dask.compute(*objs)call. - Use
total.visualize()to draw the graph (requires graphviz).
Delayed objects support attribute access, indexing and arithmetic, which become tasks too. Combine them with collections via dd.from_delayed or da.from_delayed.
Example
import threading
import time
import dask
threads_used = set()
@dask.delayed
def fetch(i):
threads_used.add(threading.current_thread().name)
time.sleep(0.5) # pretend this is slow I/O
return i * 10
@dask.delayed
def combine(values):
return sum(values)
parts = [fetch(i) for i in range(4)] # nothing runs yet
total = combine(parts)
print(total.compute(num_workers=4)) # 60
print(len(threads_used) > 1)
# True: the fetches ran on several threads at the same timeRemember
@dask.delayedmakes calls lazy; passing Delayed objects builds a graph- Independent calls run in parallel when you
compute() - Keep tasks coarse (around 100 ms or more) and side-effect free
Watch out for
- Writing
delayed(f(x)), which calls f immediately instead ofdelayed(f)(x).