TopolearnModerately sized data often becomes slow or crashes only because of wasteful dtypes and row-wise code.
Start by measuring. df.info(memory_usage="deep") or df.memory_usage(deep=True) shows real memory per column; deep=True matters because Python string objects in object columns are much larger than the 8-byte pointers counted otherwise.
Shrink dtypes:
- text with few distinct values:
category - text in general: the Arrow-backed string dtype (
"string[pyarrow]", default for text in pandas 3) is more compact and faster thanobject - numbers:
pd.to_numeric(s, downcast="integer")or explicitint32/float32when the range and precision allow - booleans stored as "Y"/"N" text: convert to
bool
Read less: usecols=, dtype= and parse_dates= in read_csv; Parquet with columns= and filters=; engine="pyarrow" for faster CSV parsing.
Compute vectorized: whole-column arithmetic, .str and .dt methods, np.where/np.select, map with dicts, groupby with built-in aggregations. Avoid apply(axis=1), iterrows and Python loops. Build DataFrames in one go from lists or dicts, never by appending rows in a loop.
Groupby tips: observed=True for categorical keys, sort=False when order doesn't matter, built-in names like "sum" instead of lambdas.
Index: lookups on a sorted, unique index are fast; repeated df[df.key == x] scans are O(n) each. If you query by key repeatedly, set_index("key").sort_index() and use .loc.
eval/query: df.eval("a + b * c") and query can use numexpr to avoid temporaries on large frames; worthwhile mainly beyond ~100k rows.
Profile before optimising: %timeit, %memit or a line profiler on realistic data. Beyond a point, the next step is a different engine (see scaling beyond memory).
Example
import numpy as np
import pandas as pd
n = 500_000
rng = np.random.default_rng(0)
df = pd.DataFrame({
"user_id": rng.integers(0, 50_000, n),
"country": rng.choice(["DE", "FR", "IT", "ES"], n).astype(object),
"flag": rng.choice(["Y", "N"], n).astype(object),
"score": rng.random(n) * 100,
})
before = df.memory_usage(deep=True).sum() / 1e6
slim = df.assign(
user_id=pd.to_numeric(df["user_id"], downcast="integer"), # -> int32
country=df["country"].astype("category"),
flag=df["flag"].eq("Y"), # -> bool
score=df["score"].astype("float32"),
)
after = slim.memory_usage(deep=True).sum() / 1e6
print(f"{before:.1f} MB -> {after:.1f} MB") # several times smaller (more on 2.x)
print(slim.dtypes.astype(str).to_dict())
# vectorized vs row loop for the same feature
fast = np.where(slim["flag"], slim["score"] * 1.1, slim["score"])
print(fast[:3].round(2))Remember
- Measure with
memory_usage(deep=True)before optimising - Use category, Arrow strings, downcast numerics and real bools
- Read only needed columns with explicit dtypes; prefer Parquet
- Stay vectorized and index by sorted keys for repeated lookups
Watch out for
- Judging memory with
memory_usage()withoutdeep=True, which hides the cost of Python string objects.