TopolearnThe scheduler decides how fast your code runs and whether you can see what it is doing.
The same Dask graph can run on different schedulers:
- threads (default for DataFrame and Array): a thread pool in your process. Great when the work releases the GIL, as most NumPy and pandas operations do.
- processes: a process pool. Helps pure-Python code that holds the GIL, but every result is pickled between processes.
- synchronous: runs everything in one thread, for debugging with
pdband readable tracebacks. - distributed: the
dask.distributedscheduler, which runs on one machine or a cluster.
Pick a local one with dask.config.set(scheduler="threads") or compute(scheduler="processes").
For anything non-trivial, use the distributed scheduler even on a laptop: from dask.distributed import Client; client = Client() starts a LocalCluster of worker processes and makes it the default. You get smarter scheduling, spilling to disk under memory pressure, futures, and the dashboard (link in client.dashboard_link, usually port 8787), which shows a live task stream, worker memory and progress. It is the single best tool for understanding performance.
Client(n_workers=4, threads_per_worker=2, memory_limit="4GB") controls resources. In scripts, create the Client under if __name__ == "__main__": because worker processes import your module. Close it with client.close() or a with block.
Example
import dask
import dask.array as da
from dask.distributed import Client, LocalCluster
if __name__ == "__main__": # worker processes re-import this module
x = da.ones((4000, 4000), chunks=(1000, 1000))
with dask.config.set(scheduler="synchronous"): # debugging mode
print(x.sum().compute()) # 16000000.0
print(x.sum().compute(scheduler="processes")) # 16000000.0
with LocalCluster(n_workers=2, threads_per_worker=2, processes=True) as cluster, \
Client(cluster) as client:
print(len(client.scheduler_info()["workers"])) # 2
print("dashboard at", client.dashboard_link) # http://127.0.0.1:8787/status
print((x + x.T).mean().compute()) # 2.0 (runs on the cluster)Remember
- Threads by default; processes for GIL-bound Python; synchronous for debugging
Client()gives you the distributed scheduler and dashboard locally- Guard Client creation with
if __name__ == "__main__":in scripts
Watch out for
- Using the process scheduler for NumPy-heavy work and paying pickling costs for nothing.