TopolearnFutures handle dynamic workloads where the next task depends on earlier results, which lazy graphs can't express.
With the distributed Client you can run tasks eagerly. future = client.submit(func, *args) starts work on the cluster right away and returns a Future: a handle to a result that lives on a worker. client.map(func, iterable) submits many at once.
future.result()waits and brings the value back;client.gather(futures)does it for a list.- Passing a Future as an argument to another
submitsends the dependency without bringing data back to your machine. The worker fetches it directly. as_completed(futures)yields futures in finishing order, so you can process results or submit follow-up work as soon as each is done. That suits searches, simulations and pipelines with dynamic structure.future.status,future.exception()andfuture.cancel()manage state and errors.
Results stay in worker memory as long as a Future referencing them exists in your session. Delete references, or let them go out of scope, to free cluster memory.
client.scatter(big_object) sends data to the workers once, returning a Future you pass to many tasks instead of shipping the object with every call.
By default submit assumes pure functions: identical calls get the same key and are computed once. Pass pure=False for functions with randomness or side effects.
Example
import time
from dask.distributed import Client, as_completed
def slow_square(x):
time.sleep(0.2 * (5 - x)) # later inputs finish sooner
return x * x
if __name__ == "__main__":
with Client(n_workers=1, threads_per_worker=5, processes=False) as client:
futures = client.map(slow_square, range(5))
order = [f.result() for f in as_completed(futures)]
print(order) # [16, 9, 4, 1, 0] (finish order)
total = client.submit(sum, futures) # depends on futures, data stays on workers
print(total.result()) # 30
lookup = client.scatter({"a": 1, "b": 2})
print(client.submit(lambda d, k: d[k], lookup, "b").result()) # 2Remember
submit/mapstart work immediately and return Futures- Pass Futures into other tasks to chain work without moving data home
as_completedhandles results as they finish; delete Futures to free memory
Watch out for
- Holding millions of Futures in a list, keeping all their results pinned in cluster memory.