TopolearnPartition size and a sorted index are the two biggest levers on Dask DataFrame performance.
Partition size matters. Each partition is a pandas DataFrame processed by one task. Too small (a few MB) and the scheduler overhead dominates; too large (several GB) and workers run out of memory. Aim for roughly 100 MB to 1 GB in memory per partition. Fix bad sizes with df.repartition(partition_size="200MB") or npartitions=n, especially after a filter that removed most rows.
Divisions are Dask's knowledge of the index: if the index is sorted across partitions, df.divisions holds the boundary values, such as (0, 1000, 2000, 2999), meaning partition 0 holds index 0 to 999 and so on. When divisions are known, Dask can:
- answer
df.loc["2026-03-01":"2026-03-07"]by reading only the relevant partitions, - join or align on the index without shuffling,
- do time-series resampling and rolling windows across partition boundaries.
If divisions are unknown, df.known_divisions is False and divisions are all None.
df.set_index("timestamp") sorts the whole dataset by that column and computes divisions. It's an expensive shuffle, so do it once and save the result to Parquet; with calculate_divisions=True, read_parquet can recover divisions from the file statistics later. If the data is already sorted, pass sorted=True to skip the shuffle.
Example
import dask.dataframe as dd
import pandas as pd
pdf = pd.DataFrame({"ts": pd.date_range("2026-01-01", periods=60, freq="D"),
"v": range(60)})
df = dd.from_pandas(pdf, npartitions=6, sort=False)
print(df.known_divisions) # False
indexed = df.set_index("ts", sorted=True) # already sorted: no shuffle
print(indexed.known_divisions, indexed.npartitions) # True 6
print(indexed.divisions[:2])
# (Timestamp('2026-01-01 00:00:00'), Timestamp('2026-01-11 00:00:00'))
week = indexed.loc["2026-02-01":"2026-02-07"] # touches only 1 partition
print(week.npartitions, week.v.sum().compute()) # 1 238
small = indexed[indexed.v % 10 == 0].repartition(npartitions=1)
print(small.npartitions) # 1Remember
- Target roughly 100 MB to 1 GB of memory per partition
- Known divisions enable fast
loc, index joins and time-series ops set_indexis a costly shuffle: do it once and save to Parquet
Watch out for
- Keeping thousands of tiny partitions after a selective filter instead of repartitioning.