Group by and aggregate

Polars · Essentials · 15 min

Per-group totals, counts and averages are the most common analysis there is.

df.group_by("key").agg(...) splits the rows by key, evaluates each aggregation expression within every group, and returns one row per group. Group by several columns with group_by("a", "b").

Inside agg, any expression works:

  • pl.col("amount").sum(), .mean(), .median(), .min(), .max(), .std()
  • pl.len() for the row count, pl.col("user").n_unique() for distinct values
  • pl.col("amount").filter(pl.col("type") == "refund").sum() to aggregate a subset
  • pl.col("item").first() or pl.col("item") alone, which collects each group's values into a list

Name results with .alias() or keyword arguments, e.g. agg(total=pl.col("amount").sum()). All aggregations in one agg run in parallel.

Order is not guaranteed. Group-by output comes back in whatever order the parallel engine finishes, and in Polars 2 the streaming engine is the default, so don't rely on first-seen order. Sort the result, or pass maintain_order=True (slower) when order matters.

Compared with pandas there is no index to reset afterwards: the keys are ordinary columns in the result. If you know SQL, group_by().agg() is exactly GROUP BY with the select list inside agg.

Example

import polars as pl

sales = pl.DataFrame({
    "shop": ["A", "B", "A", "B", "A"],
    "kind": ["sale", "sale", "refund", "sale", "sale"],
    "amount": [10.0, 5.0, -3.0, 7.0, 4.0],
})

summary = (
    sales.group_by("shop")
    .agg(
        n=pl.len(),
        net=pl.col("amount").sum(),
        refunds=pl.col("amount").filter(pl.col("kind") == "refund").sum(),
        kinds=pl.col("kind").unique().sort(),
    )
    .sort("shop")                      # output order is not guaranteed
)
print(summary.rows())
# [('A', 3, 11.0, -3.0, ['refund', 'sale']), ('B', 2, 12.0, 0.0, ['sale'])]

Remember

  • group_by(keys).agg(exprs) gives one row per group
  • Any expression can be an aggregation, including filtered ones
  • Sort the result or use maintain_order=True; order is not guaranteed

Watch out for

  • Assuming groups come back in first-appearance order.

Check yourself

  1. What is the type of the column produced by df.group_by('k').agg(pl.col('v')) (no aggregation applied)?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.pola.rs/user-guide/expressions/aggregation/