TopolearnParquet is the standard file format for analytics, and PyArrow is the engine most Python tools use to read and write it.
Parquet is a compressed, columnar file format that stores the schema. The pyarrow.parquet module, imported as pq, reads and writes it.
Writing: pq.write_table(table, "data.parquet", compression="zstd"). The default codec is Snappy; Zstd usually gives smaller files at similar speed. row_group_size= controls how many rows go into each row group, the unit readers can skip.
Reading: pq.read_table("data.parquet") returns a Table. Two arguments make reads much cheaper:
columns=["a", "b"]reads only those columns from disk, since each column is stored separately.filters=[("year", "=", 2026), ("amount", ">", 0)](a list means AND; a list of lists means OR of ANDs) skips row groups whose statistics can't match and filters rows. You can also pass an expression such aspc.field("amount") > 0.
pq.read_metadata(path) and pq.read_schema(path) inspect a file without reading data, and pq.ParquetFile(path) gives finer control such as iter_batches() for streaming through large files.
Parquet preserves Arrow types including timestamps with time zones, decimals, lists and structs, and stores pandas metadata so to_pandas() can restore the index. Prefer it over CSV for anything you read more than once.
Example
import pyarrow as pa
import pyarrow.parquet as pq
t = pa.table({
"year": [2025, 2026, 2026, 2026],
"shop": ["A", "A", "B", "B"],
"amount": [10.0, -2.0, 7.5, 3.0],
})
pq.write_table(t, "sales.parquet", compression="zstd")
meta = pq.read_metadata("sales.parquet")
print(meta.num_rows, meta.num_columns) # 4 3
part = pq.read_table(
"sales.parquet",
columns=["shop", "amount"],
filters=[("year", "=", 2026), ("amount", ">", 0)],
)
print(part.to_pydict())
# {'shop': ['B', 'B'], 'amount': [7.5, 3.0]}Remember
columns=reads only what you need;filters=skips row groups and rows- Zstd compression is a good default for smaller files
- Inspect files cheaply with
read_metadataandread_schema
Watch out for
- Reading a whole wide file and selecting columns afterwards, instead of passing
columns=.