TopolearnKnowing what the optimizer does lets you write queries it can speed up and spot why one is slow.
lf.explain() prints the optimized plan as text; lf.explain(optimized=False) shows your query as written. Read the plan bottom-up: the scan is at the bottom, the final operation at the top. lf.show_graph() draws it (needs graphviz).
The main optimizations:
- Projection pushdown: only columns the query uses are read. In the plan the scan shows something like
PROJECT 2/5 COLUMNS. - Predicate pushdown: filters move as early as possible, ideally into the scan (
SELECTION:on the scan node), where Parquet statistics can skip whole row groups. - Slice pushdown:
head(n)stops reading early. - Simplification and type coercion: constant folding, cheaper equivalent expressions.
- Common subplan / subexpression elimination: a sub-query or expression used twice is computed once.
Some things block optimizations. A Python UDF (map_elements, map_batches) is opaque, so filters can't move past it. Collecting in the middle of a pipeline cuts the plan in two. Reading CSV can't skip row groups the way Parquet can.
When two outputs share work, collect them together with pl.collect_all([q1, q2]) so the shared part runs once. Turning optimizations off (collect(optimizations=...)) is useful only when debugging a suspected optimizer bug.
Example
import polars as pl
pl.DataFrame({"a": range(6), "b": list("xyxyxy"), "c": [1.0] * 6,
"d": [0] * 6}).write_parquet("t.parquet")
q = (
pl.scan_parquet("t.parquet")
.with_columns(a2=pl.col("a") * 2)
.filter(pl.col("b") == "x")
.select("a2", "b")
)
plan = q.explain()
print(plan)
# The filter on b and the column list are pushed into the Parquet scan;
# look for the scan node showing PROJECT 2/4 COLUMNS and a SELECTION.
print(q.collect()["a2"].to_list()) # [0, 4, 8]Remember
explain()shows the optimized plan; read it bottom-up- Projection and predicate pushdown move work into the scan
- Python UDFs and early collects block optimizations
Watch out for
- Putting a
map_elementsearly in the pipeline and wondering why filters are not pushed down.