TopolearnKnowing the file layout explains why some queries skip 99% of a file and others read it all.
A Parquet file is split horizontally into row groups. Inside each, every column is stored as a column chunk, which is split into pages. The footer holds the schema and, per column chunk, statistics (min, max, null count), plus offsets so readers can jump straight to the chunks they need.
This layout enables two kinds of skipping:
- Column pruning: only the requested columns' chunks are read.
- Row-group pruning: a filter like
amount > 100skips any row group whose max is 100 or less. It only helps if data is clustered; sorting by a commonly filtered column before writing makes statistics selective.
Encodings shrink data before compression: dictionary encoding for repeated values (on by default; it falls back to plain encoding if the dictionary grows too large), run-length and bit-packing for small integers and levels, and delta encodings for sorted numbers. Compression (Snappy, Zstd, LZ4, Gzip) is then applied per page.
Tuning on write with pq.write_table:
row_group_size: too small adds overhead and footer size; too large hurts parallelism and skipping. A common target is 64 to 512 MB of uncompressed data, or around a million rows.compression="zstd",compression_level=...use_dictionary=[...],write_statistics=True,sorting_columns=
pq.ParquetFile(path).metadata.row_group(i).column(j).statistics lets you inspect all of this.
Example
import pyarrow as pa
import pyarrow.parquet as pq
t = pa.table({"amount": list(range(1000)), "shop": ["A", "B"] * 500})
pq.write_table(t, "sorted.parquet", row_group_size=250) # 4 row groups
pf = pq.ParquetFile("sorted.parquet")
md = pf.metadata
print(md.num_row_groups) # 4
for i in range(md.num_row_groups):
st = md.row_group(i).column(0).statistics
print(i, st.min, st.max, end=" | ")
# 0 0 249 | 1 250 499 | 2 500 749 | 3 750 999 |
print()
col = md.row_group(0).column(1)
print(col.compression, "RLE_DICTIONARY" in col.encodings) # SNAPPY True
# amount > 900 can only match the last row group, so readers skip the other 3
print(pq.read_table("sorted.parquet", filters=[("amount", ">", 900)]).num_rows) # 99Remember
- File = row groups; row group = column chunks; chunk = pages
- Min/max statistics let readers skip row groups, if data is clustered
- Sort by common filter columns and choose sensible row-group sizes
Watch out for
- Writing thousands of tiny row groups (or files), which bloats metadata and kills scan speed.