TopolearnData lakes are folders of many Parquet files; datasets let you query them as one table while reading only what you need.
import pyarrow.dataset as ds. ds.dataset("data/", format="parquet", partitioning="hive") discovers all files under a directory (local or cloud) and presents them as one logical table, without reading any data yet.
Partitioning encodes column values in the directory path. Hive style looks like year=2026/country=NO/part-0.parquet; the year and country columns come from the path. Filters on partition columns skip whole directories, and filters on other columns use Parquet statistics to skip row groups.
Reading is lazy and selective:
dataset.to_table(columns=[...], filter=ds.field("year") == 2026)reads only matching data.dataset.scanner(...).to_batches()ordataset.to_batches()streams RecordBatches for data larger than memory.dataset.count_rows(filter=...)anddataset.head(5)are cheap checks.
Writing: ds.write_dataset(table, "out/", format="parquet", partitioning=["year"], partitioning_flavor="hive") splits data into partition folders. existing_data_behavior="overwrite_or_ignore" controls what happens if the folder exists, and max_rows_per_file caps file size.
Choose partition columns with low cardinality that you usually filter on, like date or region. Partitioning by a high-cardinality column such as user ID creates millions of tiny files and slows everything down.
Example
import pyarrow as pa
import pyarrow.dataset as ds
t = pa.table({
"year": [2025, 2025, 2026, 2026],
"country": ["NO", "SE", "NO", "SE"],
"amount": [5, 3, 8, 2],
})
ds.write_dataset(t, "lake", format="parquet", partitioning=["year"],
partitioning_flavor="hive",
existing_data_behavior="overwrite_or_ignore")
dataset = ds.dataset("lake", format="parquet", partitioning="hive")
print(sorted(f.split("lake/")[1] for f in dataset.files))
# ['year=2025/part-0.parquet', 'year=2026/part-0.parquet']
recent = dataset.to_table(columns=["country", "amount"],
filter=ds.field("year") == 2026)
print(recent.sort_by("country").to_pydict())
# {'country': ['NO', 'SE'], 'amount': [8, 2]}
print(dataset.count_rows(filter=ds.field("amount") > 2)) # 3Remember
- A dataset is many files presented as one lazy table
- Hive partitions turn path segments into columns and enable directory skipping
- Partition on low-cardinality columns you often filter by
Watch out for
- Partitioning by a high-cardinality column and creating thousands of tiny files.