TopolearnArrow IPC is the fastest way to move tables between processes or save them for quick reload, because no parsing or conversion happens.
Arrow IPC writes the in-memory columnar layout to bytes almost unchanged, so reading it back needs no parsing: just pointers into a buffer.
There are two flavours:
- The stream format (
pa.ipc.new_stream(sink, schema)andpa.ipc.open_stream(source)) is a sequence of record batches of unknown length, ideal for sockets, pipes and HTTP bodies. - The file format (
pa.ipc.new_fileandpa.ipc.open_file) adds a footer, so readers can jump to any batch. Its files usually use the.arrowextension.
Feather v2 is the IPC file format with optional LZ4 or Zstd compression, via pyarrow.feather.write_feather(table, path) and read_feather(path) or read_table(path).
Memory mapping makes reads nearly free. pa.memory_map(path) maps the file into virtual memory, and pa.ipc.open_file(mm).read_all() builds a Table whose buffers point straight into the mapped file: nothing is copied, and the OS loads only pages you touch. This only works for uncompressed IPC files; compressed data must be decompressed into new memory.
Compared with Parquet: IPC and Feather are faster to read and write but larger, and aimed at short-term storage and interchange. Parquet is smaller and better for long-term storage and analytics engines.
Example
import pyarrow as pa
t = pa.table({"id": list(range(100_000)), "x": [1.5] * 100_000})
with pa.OSFile("data.arrow", "wb") as sink:
with pa.ipc.new_file(sink, t.schema) as writer:
for batch in t.to_batches(max_chunksize=25_000):
writer.write_batch(batch)
before = pa.total_allocated_bytes()
with pa.memory_map("data.arrow", "r") as source:
reader = pa.ipc.open_file(source)
print(reader.num_record_batches) # 4
loaded = reader.read_all()
print(loaded.num_rows) # 100000
print(pa.total_allocated_bytes() - before) # 0 (buffers point into the map)
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, t.schema) as w:
w.write_table(t.slice(0, 3))
print(pa.ipc.open_stream(sink.getvalue()).read_all()["id"].to_pylist()) # [0, 1, 2]Remember
- IPC = Arrow's memory layout on the wire or disk, no parsing needed
- Stream format for pipes and sockets; file format for random access
- Memory-mapped uncompressed IPC files load with zero copies
Watch out for
- Expecting zero-copy reads from a compressed Feather file; decompression allocates memory.