TopolearnProduction Dask means picking a deployment that fits your infrastructure and keeping software identical on every machine.
Every deployment starts the same three pieces, a scheduler, workers and your client, through different cluster managers with a shared interface: cluster.scale(n), cluster.adapt(minimum=, maximum=), Client(cluster).
- Local:
LocalCluster, one machine. - Kubernetes: the
dask-kubernetesoperator withKubeCluster, or a Helm chart for a long-lived cluster. - HPC:
dask-jobqueue(SLURMCluster,PBSCluster, etc.) submits workers as batch jobs. - Cloud VMs:
dask-cloudprovider(EC2, Azure, GCP) or managed services such as Coiled. - Manual: run
dask scheduleranddask worker tcp://host:8786yourself, or useSSHCluster.
Adaptive scaling (cluster.adapt(minimum=2, maximum=50)) lets the scheduler request more workers when work is queued and release idle ones, saving money on bursty workloads.
Pitfalls specific to clusters:
- Environment drift: the client, scheduler and workers must have the same versions of Dask, Python and every library your tasks use.
client.get_versions(check=True)checks. Use one container image or environment everywhere. - Data access: workers read files themselves, so paths must be reachable from every worker (object storage or shared filesystems, not your laptop's disk).
- Security: enable TLS and avoid exposing the scheduler port publicly.
- Configuration lives in YAML or environment variables like
DASK_DISTRIBUTED__WORKER__MEMORY__TARGET, readable viadask.config.get.
Example
import dask
from dask.distributed import Client, LocalCluster
if __name__ == "__main__":
# The same pattern works with KubeCluster, SLURMCluster, EC2Cluster...
with LocalCluster(n_workers=0, threads_per_worker=1, processes=False) as cluster:
cluster.adapt(minimum=1, maximum=3) # scale with demand
with Client(cluster) as client:
client.wait_for_workers(1)
print(len(client.scheduler_info()["workers"]) >= 1) # True
versions = client.get_versions(check=True) # raises on mismatch
print("python" in versions["scheduler"]["host"]) # True
print(dask.config.get("distributed.scheduler.allowed-failures")) # 3
print(client.submit(sum, [1, 2, 3]).result()) # 6Remember
- Cluster managers share one interface: scale, adapt, Client(cluster)
- Keep identical environments on client, scheduler and workers
- Workers read data themselves: use storage every worker can reach
Watch out for
- Reading local file paths that exist only on the client machine from remote workers.