Topolearnapply is the most common cause of slow pandas code; knowing the alternatives often gives 100x speedups.
apply calls a Python function repeatedly: s.apply(f) once per element, df.apply(f) once per column, and df.apply(f, axis=1) once per row, building a Series for each row. That's flexible and easy to write, but it's a Python-level loop. On a million rows, axis=1 apply can take tens of seconds where a vectorized expression takes milliseconds.
Prefer, in this order:
- Column arithmetic and ufuncs:
df.a * df.b,np.log(df.x),df.x.clip(0, 100). - Built-in methods:
.str,.dt,round,abs,cumsum,diff,shift,between,isin. - Conditional logic:
np.where,np.select,Series.where/mask,pd.cut. - Lookups:
s.map(dict_or_series);mergefor multi-column lookups. - Group-wise work: groupby with built-in aggregations or
transform.
s.map(func) and s.apply(func) on a Series are equivalent loops; map with a dict is a fast hash lookup. DataFrame.map (formerly applymap, renamed in 2.1) applies element-wise to every cell, also a loop.
Legitimate uses of apply remain: calling a library function that only accepts scalars (a geocoder, a parser), complex per-group logic in groupby(...).apply, or small data where clarity matters more than speed. When you do need a loop over rows, itertuples() is much faster than iterrows(), which builds a Series per row and loses dtypes.
If a computation truly can't be vectorized and the data is large, push the loop into compiled code with Numba, or rethink the algorithm. Measure first: %timeit on a realistic sample tells you whether it matters.
Example
import numpy as np
import pandas as pd
import time
rng = np.random.default_rng(0)
df = pd.DataFrame({"price": rng.uniform(1, 100, 200_000),
"qty": rng.integers(0, 10, 200_000)})
def tier_row(row): # row-wise Python logic
total = row["price"] * row["qty"]
return "big" if total > 300 else "small" if total > 0 else "none"
t = time.perf_counter()
slow = df.apply(tier_row, axis=1)
t_apply = time.perf_counter() - t
t = time.perf_counter()
total = df["price"] * df["qty"]
fast = pd.Series(np.select([total > 300, total > 0], ["big", "small"], "none"), index=df.index)
t_vec = time.perf_counter() - t
print(slow.equals(fast)) # True
print(f"apply {t_apply:.2f}s, vectorized {t_vec:.3f}s") # typically 30-100x apart
codes = pd.Series(["N", "S", "N"])
print(codes.map({"N": "North", "S": "South"}).tolist()) # fast dict lookupRemember
apply(axis=1)is a Python loop over rows and is slow on large data- Prefer column arithmetic, built-in methods,
np.where/np.selectandmapwith dicts - Use
itertuplesrather thaniterrowsif you must loop - apply is fine for small data or scalar-only library calls
Watch out for
- Reaching for
apply(lambda row: ..., axis=1)for simple arithmetic between columns.