TopolearnMost Dask DataFrame code is ordinary pandas code; knowing the gaps keeps you from slow or failing patterns.
Dask DataFrame implements a large part of the pandas API, and each call becomes per-partition pandas work plus a combine step.
Works just like pandas: column selection, boolean filtering df[df.amount > 0], assign, arithmetic, .str and .dt accessors, fillna, astype, isin, and value_counts.
Reductions and groupby are done as a tree: aggregate inside each partition, then combine. df.groupby("shop").amount.sum(), .agg({"amount": ["sum", "mean"], "id": "count"}) and nunique all work and are efficient when the number of groups is small. With many groups, pass split_out=n to spread the result over several partitions. groupby().apply(f) needs all rows of a group in one place, so it triggers a full shuffle; prefer built-in aggregations.
Differences from pandas:
- There is no positional
ilocrow access, andlocneeds a sorted index (see the partitions lesson). - Some operations need to know the output dtypes up front, so they take a
meta=argument. sort_valuesandset_indexare expensive global operations.- Results are lazy; call
compute(), orpersist()to keep intermediate data in memory.
Example
import dask.dataframe as dd
import pandas as pd
pdf = pd.DataFrame({
"shop": ["A", "B", "A", "C", "B", "A"],
"amount": [10.0, -2.0, 5.0, 7.0, 3.0, None],
"items": [2, 1, 1, 3, 1, 2],
})
df = dd.from_pandas(pdf, npartitions=2)
clean = df[df.amount > 0].assign(per_item=lambda d: d.amount / d["items"])
summary = clean.groupby("shop").agg(
total=("amount", "sum"), n=("amount", "count"), avg_item=("per_item", "mean")
)
print(summary.compute().sort_index())
# total n avg_item
# shop
# A 15.0 2 5.00
# B 3.0 1 3.00
# C 7.0 1 2.33...
print(df.shop.value_counts().compute().sort_index().tolist()) # [3, 2, 1]Remember
- Most pandas column, filter and aggregation code runs unchanged
- Groupby aggregations are tree reductions;
groupby().applyshuffles - Avoid row-positional access; it doesn't fit the partitioned model
Watch out for
- Using
groupby().applywith a custom function where a built-in aggregation would do.