TopolearnKnowing what the optimizer can and can't rewrite helps you write code that it speeds up automatically.
Since 2025, Dask DataFrame is built on an expression system (originally the dask-expr project). Each operation creates an expression node, and before running, Dask optimizes the whole expression tree, much like a database query planner.
Optimizations include:
- Projection pushdown: only columns used later are read from Parquet or kept through the pipeline.
- Predicate pushdown: filters move toward the reader, where Parquet filters can skip row groups.
- Partition squashing: after
read_parquet, small files may be combined into fewer partitions automatically. - Merge and groupby planning: choosing broadcast joins or tree reductions from size estimates.
- Blockwise fusion: chains of per-partition operations are fused into single tasks, which shrinks the graph.
Inspect it with df.pprint() for the raw expression tree, df.optimize().pprint() for the optimized one (fuse=False keeps fused steps readable), and df.explain() for a graphviz drawing (needs the graphviz package).
The optimizer works best when it can see the whole query. Opaque steps limit it: map_partitions with arbitrary functions, calling compute() midway, or converting to delayed objects. Write the query fully, then compute once.
Example
import dask.dataframe as dd
import pandas as pd
pd.DataFrame({"a": range(10), "b": list("xyxyxyxyxy"),
"c": [1.0] * 10, "d": ["unused"] * 10}).to_parquet("t.parquet")
df = dd.read_parquet("t.parquet")
q = df[df.a > 2].groupby("b").a.sum()
q.optimize(fuse=False).pprint()
# Sum(TreeReduce): split_every=8
# Sum(GroupByChunk): chunk=<methodcaller: sum> columns=a ...
# Filter:
# FusedIO:
# ReadParquetFSSpec: path='t.parquet' columns=['a', 'b']
# GT: right=2
# Projection: columns='a'
# FusedIO:
# ReadParquetFSSpec: path='t.parquet' columns=['a', 'b']
print(q.compute().sort_index().to_dict()) # {'x': 18, 'y': 24}Remember
- Dask DataFrame optimizes the whole expression before running
- Projection and predicate pushdown reach into
read_parquet optimize().pprint()shows the plan; opaque steps limit optimization
Watch out for
- Calling
.compute()or.persist()mid-pipeline 'to be safe', which splits the query the optimizer could have improved.