TopolearnGroup-level summaries (per customer, per month, per product) are the backbone of analysis and reporting.
`groupby` follows the split-apply-combine pattern: split rows by key, apply a function to each group, combine the results. It's the pandas counterpart to SQL's GROUP BY.
df.groupby("region")["sales"].sum()The result is indexed by the group keys. Group by several columns with a list; the result then has a MultiIndex, or pass as_index=False (or call reset_index()) to keep keys as ordinary columns.
For several statistics, use named aggregation, which produces clean, flat column names:
df.groupby("region").agg(
total=("sales", "sum"),
orders=("order_id", "count"),
avg_price=("price", "mean"),
)Know the difference between three kinds of operation:
- aggregate (
sum,mean,agg): one row per group - transform: a result the same length as the input, aligned with the original rows.
df.groupby("region")["sales"].transform("mean")puts each row's group mean next to it, ready forsales / group_mean. - filter: keeps or drops whole groups, e.g.
.filter(lambda g: len(g) >= 10)
Other details worth knowing: size() counts rows per group including missing values, while count() counts non-missing values per column. Rows whose key is NaN are dropped unless you pass dropna=False. For categorical keys, observed=True (the default from pandas 3.0) shows only categories that actually occur.
Built-in aggregation names ("sum", "mean", "max", "nunique", "first") run in optimised code. A custom Python function via agg(lambda ...) or apply runs once per group in Python and can be dramatically slower with many groups.
Example
import pandas as pd
df = pd.DataFrame({
"region": ["N", "S", "N", "S", "N", "E"],
"rep": ["ana", "bo", "ana", "di", "cy", "ed"],
"sales": [100, 250, 50, 80, 120, 60],
})
print(df.groupby("region")["sales"].sum().to_dict()) # {'E': 60, 'N': 270, 'S': 330}
summary = df.groupby("region", as_index=False).agg(
total=("sales", "sum"),
reps=("rep", "nunique"),
best=("sales", "max"),
)
print(summary)
# region total reps best
# 0 E 60 1 60
# 1 N 270 2 120
# 2 S 330 2 250
df["share"] = df["sales"] / df.groupby("region")["sales"].transform("sum")
print(df["share"].round(2).tolist()) # [0.37, 0.76, 0.19, 0.24, 0.44, 1.0]
big = df.groupby("region").filter(lambda g: len(g) >= 2)
print(sorted(big["region"].unique())) # ['N', 'S']Remember
- groupby = split by key, apply, combine; keys become the index unless
as_index=False - Named aggregation
agg(new=('col', 'func'))gives tidy output transformreturns a row-aligned result;filterkeeps whole groups- Built-in aggregation names are fast; Python lambdas per group are slow
Watch out for
- Forgetting that NaN keys are dropped by default, so group totals don't add up to the overall total.