TopolearnChains avoid half-modified intermediate variables and make data pipelines easy to read, test and reorder.
Most pandas methods return a new DataFrame, so they can be chained: each step takes the previous result. Wrap the chain in parentheses and put one step per line:
result = (
df
.query("status == 'paid'")
.assign(total=lambda d: d.price * d.qty)
.groupby("region", as_index=False)["total"].sum()
.sort_values("total", ascending=False)
)The building blocks that make chaining work:
- `assign` with lambdas, so new columns refer to the DataFrame at that step (
d), not a stale outer variable - `query` or `loc[lambda d: d.x > 0]` for filtering mid-chain
- `rename`,
drop,astype,set_axis,reset_indexfor tidying - **
pipe(func, *args)** to insert your own function that takes and returns a DataFrame; this is how you name and reuse steps likeclean_namesordrop_test_accounts
Benefits: no df2, df3, df_final variables; no accidental reuse of a half-processed frame; each step is easy to comment out while debugging. It also works naturally with copy-on-write, because you never modify an object that another variable still refers to.
Costs and remedies: a long chain is harder to inspect. Insert .pipe(lambda d: (print(d.shape), d)[1]) to peek at intermediate shapes, or break the chain into a few well-named stages. Don't force everything into one expression; a chain of 5-10 steps per stage is plenty.
Avoid inplace=True in chains (and in general): it returns None, breaks chaining, and despite its name usually doesn't save memory.
Example
import pandas as pd
raw = pd.DataFrame({
"Region ": ["N", "S", "N", "S", "N"],
"Status": ["paid", "paid", "refund", "paid", "paid"],
"Price": [10.0, 4.0, 10.0, 6.5, 3.0],
"Qty": [2, 5, 1, 2, 10],
})
def clean_names(d):
return d.rename(columns=lambda c: c.strip().lower())
report = (
raw
.pipe(clean_names)
.query("status == 'paid'")
.assign(total=lambda d: d.price * d.qty)
.loc[lambda d: d.total > 10]
.groupby("region", as_index=False)
.agg(revenue=("total", "sum"), orders=("total", "size"))
.sort_values("revenue", ascending=False)
)
print(report)
# region revenue orders
# 0 N 50.0 2
# 1 S 33.0 2Remember
- Wrap chains in parentheses, one method per line
assign(col=lambda d: ...)andloc[lambda d: ...]refer to the current steppipeslots your own DataFrame-to-DataFrame functions into a chain- Avoid
inplace=True: it returns None and rarely saves memory
Watch out for
- Referring to the original variable inside a chain (
df.priceinstead oflambda d: d.price), which uses stale, unfiltered data.