TopolearnThe physical layout is what makes Arrow zero-copy across languages, and understanding it explains memory use and interop behaviour.
Every Arrow array is a few flat buffers plus metadata (type, length, null count, offset), defined by a language-independent specification.
- Validity bitmap: one bit per slot, 1 = valid, 0 = null. It may be omitted entirely when there are no nulls.
- Fixed-width types (ints, floats, timestamps): one values buffer,
length * widthbytes. Null slots still occupy space with unspecified contents. - Variable-size binary and strings: an offsets buffer of
length + 1int32 values plus a data buffer of concatenated bytes; valueiisdata[offsets[i]:offsets[i+1]].large_stringuses int64 offsets for more than 2 GB of data. Newer string view types store 16-byte views with short strings inline. - Lists: offsets into a child array. Structs: one child array per field. Dictionary arrays: indices plus a dictionary array.
- Booleans are bit-packed values.
Buffers are aligned (64-byte alignment is recommended) for SIMD. A slice doesn't copy; it records an offset and length into the same buffers, which is why arr.offset can be non-zero and why serialising a small slice of a huge array can still carry the big buffers unless you compact it.
arr.buffers() returns the raw buffers so you can see this layout directly. Same bytes, same meaning in C++, Rust, Java, Go and JavaScript: that's what lets a Polars, DuckDB or Spark process hand data to Python without conversion.
Example
import pyarrow as pa
s = pa.array(["hi", None, "arrow"])
validity, offsets, data = s.buffers()
print(bin(validity.to_pybytes()[0])) # 0b101 (slot 1 is null)
print(pa.Array.from_buffers(pa.int32(), 4, [None, offsets]).to_pylist())
# [0, 2, 2, 7] value i = data[offsets[i]:offsets[i+1]]
print(data.to_pybytes()[:7]) # b'hiarrow'
ints = pa.array([1, 2, 3], type=pa.int32())
print(ints.buffers()[0], ints.buffers()[1].size) # None 12 (no nulls: no bitmap)
sl = ints.slice(1, 2)
print(sl.offset, sl.buffers()[1].address == ints.buffers()[1].address) # 1 TrueRemember
- Arrays = validity bitmap + values buffer (+ offsets / children)
- Strings and lists use offsets into a shared data buffer
- Slices share buffers and record an offset
Watch out for
- Serialising a tiny slice of a huge array and shipping the whole underlying buffers.