TopolearnWhen data outgrows memory, streaming lets one machine do work that would otherwise need a cluster.
The streaming engine, the default for collect() in Polars 2, splits data into small batches called morsels and pushes them through the plan on all cores. Operators like filter, select and with_columns are naturally streaming. Group-bys, joins and sorts keep state, and spill or partition as needed, so most queries run in memory bounded well below the input size.
To avoid ever holding the result, write it directly with a sink: lf.sink_parquet("out.parquet"), sink_csv, sink_ipc, sink_ndjson. Sinks can write partitioned output, for example pl.PartitionBy("out/", key="country"), producing hive-style directories. To consume results piece by piece in Python, iterate lf.collect_batches().
Inputs scale the same way. pl.scan_parquet("data/**/*.parquet", hive_partitioning=True) reads directory trees like year=2026/month=03/ and turns the path segments into columns, so a filter on year skips whole directories. Cloud URLs (s3://, gs://, az://) work with storage_options.
Things that defeat streaming: Python UDFs that need the whole column, operations that need global order (some over and rolling patterns), and pulling a huge result back with collect(). Check with lf.explain() and keep memory flat by finishing with a sink.
Tune with pl.Config.set_streaming_chunk_size only after measuring; the defaults are usually right.
Example
import os
import polars as pl
os.makedirs("raw", exist_ok=True)
for i in range(3): # pretend these are huge files
pl.DataFrame({"country": ["NO", "SE"] * 2, "amount": [i, i + 1, i + 2, i + 3]}) \
.write_parquet(f"raw/part{i}.parquet")
q = (
pl.scan_parquet("raw/*.parquet")
.filter(pl.col("amount") > 1)
.group_by("country")
.agg(total=pl.col("amount").sum())
)
q.sink_parquet("totals.parquet") # result streamed straight to disk
print(pl.read_parquet("totals.parquet").sort("country").rows())
# [('NO', 11), ('SE', 17)]
for batch in pl.scan_parquet("raw/*.parquet").collect_batches():
print(batch.height, end=" ") # batches arrive one by one, e.g. 4 4 4Remember
- Polars 2 collects with the streaming engine: batched, multi-core, bounded memory
- Finish with
sink_*so the result never has to fit in RAM - Hive-partitioned scans skip directories when you filter on partition columns
Watch out for
- Streaming a huge query and then calling
collect()on a result that is itself too large for memory.