TopolearnAt scale, memory, not CPU, is usually the limit, and knowing who owns which buffer prevents leaks and surprise copies.
Arrow allocates buffers from a MemoryPool. pa.default_memory_pool() is usually mimalloc or jemalloc, depending on the platform; the ARROW_DEFAULT_MEMORY_POOL environment variable chooses one (system, jemalloc, mimalloc). pa.total_allocated_bytes() and pool.max_memory() report usage, and many functions accept memory_pool= so you can track one operation in isolation.
Memory is reference-counted. A buffer is freed when the last array, table or slice referencing it is gone. Two consequences:
- Keeping a tiny slice of a big table alive keeps the whole big buffer alive.
- Allocators often keep freed memory for reuse, so the process's RSS may not drop right away.
pool.release_unused()asks the allocator to return memory to the OS.
What copies and what doesn't:
- Zero-copy: slicing, selecting columns,
Table.from_batches,concat_tables(chunks are reused), IPC reads from memory maps,pa.py_buffer(bytes_obj), NumPy numeric conversion without nulls. - Copies:
combine_chunks(), casts that change representation, compute results, filtering, pandas conversion of strings or nulls, decompression.
pa.foreign_buffer(address, size, base) wraps memory owned by another library while keeping base alive. That is the low-level mechanism behind zero-copy interop.
Example
import pyarrow as pa
pool = pa.default_memory_pool()
print(pool.backend_name in {"mimalloc", "jemalloc", "system"}) # True
before = pool.bytes_allocated()
big = pa.array(range(1_000_000), type=pa.int64())
print(pool.bytes_allocated() - before >= 8_000_000) # True (8 bytes x 1M)
small = big.slice(0, 10) # zero-copy view
del big
print(pool.bytes_allocated() - before >= 8_000_000) # True: the slice pins the buffer
compact = pa.concat_arrays([small]) # copies just the 10 values
del small
print(pool.bytes_allocated() - before < 1_000) # True: big buffer freed
raw = b"\x01\x00\x00\x00\x02\x00\x00\x00"
view = pa.Array.from_buffers(pa.int32(), 2, [None, pa.py_buffer(raw)])
print(view.to_pylist()) # [1, 2] (wraps the bytes, no copy)Remember
- Allocations come from a MemoryPool you can inspect and choose
- Buffers are reference-counted; slices keep parents alive
- Know which operations share buffers and which allocate
Watch out for
- Holding a small slice of a huge table in a cache and wondering why memory never drops.