TopolearnCategoricals cut memory dramatically, speed up groupby, and encode meaningful order like low < mid < high.
A categorical column stores each distinct value once (the categories) and keeps a small integer code per row pointing to its category. A column of a million country names with 200 distinct values shrinks from tens of megabytes of Python strings to about a megabyte of int8/int16 codes plus the category list.
Convert with astype("category"), or read directly with dtype={"country": "category"}. Inspect with s.cat.categories and s.cat.codes.
Ordered categories carry meaning beyond memory:
size = pd.CategoricalDtype(["S", "M", "L", "XL"], ordered=True)
df["size"] = df["size"].astype(size)Now sort_values follows S < M < L < XL instead of alphabetical order, and comparisons like df["size"] >= "L" work. pd.cut and pd.qcut produce ordered categoricals for numeric bins.
Things that behave differently:
- Values not in the category list become NaN on conversion, and assigning a new, unknown value raises. Use
s.cat.add_categories,rename_categories,remove_unused_categoriesorset_categoriesto change the list. - In
groupby, categories that don't occur can show up as empty groups unlessobserved=True. pandas 3.0 makesobserved=Truethe default; in 2.x pass it explicitly to avoid surprising rows (and a FutureWarning). - Concatenating categoricals with different category lists falls back to
object; usepd.api.types.union_categoricalsif needed.
Use categoricals for low-cardinality text (status, country, product line), not for mostly-unique values like IDs or free text, where the category list is as big as the data. Parquet preserves the dtype; CSV doesn't.
Example
import pandas as pd
n = 300_000
status = pd.Series(["active", "paused", "closed"] * (n // 3))
print(status.memory_usage(deep=True) // 1024, "KiB as text")
cat = status.astype("category")
print(cat.memory_usage(deep=True) // 1024, "KiB as category") # far smaller
print(cat.cat.categories.tolist(), cat.cat.codes[:3].tolist()) # ['active', 'closed', 'paused'] [0, 2, 1]
sizes = pd.CategoricalDtype(["S", "M", "L", "XL"], ordered=True)
df = pd.DataFrame({"size": ["L", "S", "XL", "M"], "qty": [3, 5, 1, 2]})
df["size"] = df["size"].astype(sizes)
print(df.sort_values("size")["size"].tolist()) # ['S', 'M', 'L', 'XL']
print(df.loc[df["size"] >= "L", "qty"].sum()) # 4
print(df.groupby("size", observed=False)["qty"].sum().to_dict()) # all 4 categories
bins = pd.cut(pd.Series([3, 17, 45, 80]), [0, 18, 65, 120], labels=["kid", "adult", "senior"])
print(bins.tolist()) # ['kid', 'kid', 'adult', 'senior']Remember
- Categoricals store integer codes plus a list of categories
- Big memory and groupby wins for low-cardinality text
- Ordered categories sort and compare in a meaningful order
- Pass
observed=Truewhen grouping by categoricals (default in pandas 3)
Watch out for
- Assigning a value that isn't among the categories and getting an error (or NaN on conversion).