TopolearnOne careless Python lambda can make a Polars query 10 to 100 times slower; experts know when to avoid them and what to use instead.
Polars runs expressions in Rust on Arrow buffers across many threads. Python code breaks that model:
- `map_elements(f, return_dtype=...)` calls a Python function once per value. It holds the GIL, runs single-threaded, converts each value to a Python object, and is opaque to the optimizer. It is the last resort. Polars even warns when it recognizes a lambda it could translate into a native expression.
- `map_batches(f)` calls the function once per whole Series (or batch), so a NumPy-vectorized function stays fast. NumPy ufuncs can be applied directly:
np.log(pl.col("x"))dispatches without leaving the engine. Batch-wise functions must not depend on how data is chunked unless you account for it. - `pl.struct(...).map_batches` passes several columns at once.
Before writing a UDF, look for a native path: the str, list, dt and struct namespaces, pl.when, list.eval, fold/reduce across columns, or replace and replace_strict for mappings.
When you genuinely need custom logic at native speed, write an expression plugin: a Rust function compiled with pyo3-polars and registered with polars.plugins.register_plugin_function. Plugins run in parallel without the GIL and appear in query plans like built-ins. Always set return_dtype on Python UDFs so the lazy schema is known.
Example
import time
import numpy as np
import polars as pl
df = pl.DataFrame({"x": np.arange(1, 1_000_001, dtype=np.float64)})
t = time.perf_counter()
slow = df.select(pl.col("x").map_elements(lambda v: v * 2 + 1,
return_dtype=pl.Float64))
t_slow = time.perf_counter() - t
t = time.perf_counter()
fast = df.select(pl.col("x") * 2 + 1) # native expression
t_fast = time.perf_counter() - t
batch = df.select(pl.col("x").map_batches(np.log, return_dtype=pl.Float64))
print(slow.equals(fast), round(batch["x"][0], 3)) # True 0.0
print(f"native ~{t_slow / t_fast:.0f}x faster") # often 10-100x, machine-dependentRemember
map_elements= per-value Python: slow, single-threaded, opaquemap_batcheswith vectorized NumPy code is acceptable- Rust expression plugins give custom logic at native speed
Watch out for
- Using
map_elementsfor logic that a namespace method,when/thenorreplacealready covers.