TopolearnPyArrow's multi-threaded CSV reader is one of the fastest in Python and backs pandas' engine='pyarrow'.
pyarrow.csv.read_csv(path) parses CSV with multiple threads straight into a Table. It reads compressed files like .csv.gz automatically. Behaviour is controlled by three option objects:
ReadOptions(skip_rows=..., column_names=[...], block_size=...)for the raw input.ParseOptions(delimiter=";", quote_char='"', newlines_in_values=True)for the CSV dialect.ConvertOptions(column_types={"zip": pa.string()}, null_values=["", "NA"], include_columns=[...], timestamp_parsers=["%d/%m/%Y"])for turning text into typed columns.
Type inference looks at the data, so give explicit column_types for IDs and codes with leading zeros, and for columns that might look numeric in the first block but contain text later.
For files too large for memory, pyarrow.csv.open_csv(path) returns a streaming reader that yields RecordBatches. To write, use pyarrow.csv.write_csv(table, path).
pyarrow.json.read_json(path) reads newline-delimited JSON (one object per line), inferring nested objects as structs and arrays as lists. Pass ParseOptions(explicit_schema=...) to fix the types. A single big JSON array isn't supported by this reader; convert it to one object per line first.
Example
import pyarrow as pa
from pyarrow import csv, json
with open("people.csv", "w") as f:
f.write("id;zip;joined;score\n1;02134;2026-01-05;9.5\n2;10001;2026-02-11;NA\n")
t = csv.read_csv(
"people.csv",
parse_options=csv.ParseOptions(delimiter=";"),
convert_options=csv.ConvertOptions(
column_types={"zip": pa.string()},
null_values=["NA"],
),
)
print(t.schema.types)
# [DataType(int64), DataType(string), DataType(date32[day]), DataType(double)]
print(t["zip"].to_pylist()) # ['02134', '10001']
with open("events.jsonl", "w") as f:
f.write('{"user": "ana", "tags": ["a", "b"]}\n{"user": "ben", "tags": []}\n')
print(json.read_json("events.jsonl").schema)
# user: string
# tags: list<item: string>
# child 0, item: stringRemember
- Three option objects: ReadOptions, ParseOptions, ConvertOptions
- Set
column_typesfor IDs and codes to keep leading zeros read_jsonexpects newline-delimited JSON
Watch out for
- Passing a JSON file containing one big array to
read_json; it expects one object per line.