TopolearnEvery Dask API follows this model, so understanding it explains both Dask's power and its surprises.
Dask scales familiar tools by splitting the work into many small pieces:
- A Dask DataFrame is a collection of pandas DataFrames, called partitions, split along the rows.
- A Dask Array is a grid of NumPy arrays called chunks.
- dask.delayed and futures wrap arbitrary Python functions.
Operations are lazy. df.x.sum() doesn't compute anything; it records tasks in a task graph: sum each partition, then combine the partial sums. Calling .compute() hands the graph to a scheduler, which runs independent tasks in parallel on threads, processes or a cluster and returns a normal pandas or NumPy result.
This gives two benefits: parallelism (many cores work at once) and out-of-core processing (only a few partitions need to be in memory at a time, so data can exceed RAM).
Rules of thumb:
- Call
compute()once at the end, not after every step. Computing several results together, as indask.compute(a, b), shares the work they have in common. - The result of
compute()must fit in memory. Reduce first (aggregate, filter), or write results to files. - If your data fits comfortably in memory and pandas is fast enough, plain pandas is simpler. Dask pays off when data or compute time grow.
Example
import dask
import dask.dataframe as dd
import pandas as pd
pdf = pd.DataFrame({"x": range(10), "y": [1.0] * 10})
df = dd.from_pandas(pdf, npartitions=3) # 3 pandas pieces
print(df.npartitions) # 3
total = df.x.sum() # lazy: nothing computed yet
print(type(total).__name__) # Scalar
print(total.compute()) # 45
mean_x, max_y = dask.compute(df.x.mean(), df.y.max()) # one shared pass
print(mean_x, max_y) # 4.5 1.0Remember
- Dask collections are many pandas or NumPy pieces plus a lazy task graph
compute()runs the graph in parallel and returns in-memory results- Compute once, and compute related results together with
dask.compute
Watch out for
- Calling
.compute()on a huge un-reduced DataFrame and running out of memory.