TopolearnMemory layout explains Polars' performance, its interop costs, and many surprises around copies and types.
Each Polars Series is a ChunkedArray: a list of Arrow arrays, each with a values buffer and a validity bitmap marking nulls. Concatenating or reading many files appends chunks rather than copying. Many chunks slow down some operations, so df.rechunk() merges them into one contiguous buffer; df.n_chunks() shows the count.
Strings use Arrow's string view layout: short strings sit inline in a 16-byte view, longer ones point into shared buffers, which makes slicing and filtering strings cheap. For repeated labels use pl.Categorical (a dictionary of strings plus integer codes) or pl.Enum([...]) when the categories are fixed and known; both save memory and speed up group-bys and joins. In Polars 2, categoricals are always ordered lexically when sorted.
Interop follows the Arrow layout:
df.to_arrow()andpl.from_arrow(table)are usually zero-copy.s.to_numpy()is zero-copy for numeric columns without nulls and a single chunk; otherwise it copies (nulls become NaN in floats).df.to_pandas()copies by default;use_pyarrow_extension_array=Truekeeps Arrow-backed columns.- Polars supports the Arrow PyCapsule interface (
__arrow_c_stream__), so libraries like DuckDB and PyArrow exchange data with no copy. The old DataFrame interchange protocol was removed in Polars 2.
df.estimated_size("mb") reports memory use so you can check the effect of types and rechunking.
Example
import polars as pl
import pyarrow as pa
a = pl.DataFrame({"x": [1, 2, 3]})
b = pl.DataFrame({"x": [4, 5]})
df = pl.concat([a, b], rechunk=False)
print(df.n_chunks()) # 2 (no copy was made)
print(df.rechunk().n_chunks()) # 1
labels = pl.Series("city", ["Oslo, Norway (capital)", "Lima, Peru (capital)"] * 150_000)
enum = pl.Enum(["Lima, Peru (capital)", "Oslo, Norway (capital)"])
print([round(labels.cast(t).estimated_size("kb")) for t in (pl.String, pl.Categorical, enum)])
# [6152, 1172, 293] (codes + one dictionary beat repeated strings)
tbl = df.rechunk().to_arrow() # zero-copy Arrow table
print(isinstance(tbl, pa.Table), tbl.num_rows) # True 5
arr = df.rechunk()["x"].to_numpy() # zero-copy: numeric, no nulls, 1 chunk
print(arr.flags.writeable) # False (it views Polars' memory)Remember
- A Series is a list of Arrow chunks;
rechunk()makes it contiguous - Categorical and Enum shrink repeated strings
- Arrow and PyCapsule exchange is zero-copy; pandas conversion usually copies
Watch out for
- Writing to a NumPy array obtained from
to_numpy()and expecting it to be a private copy.