TopolearnIt is the reason pandas, Polars, DuckDB and PyArrow can hand each other tables instantly, and how you make your own library join that ecosystem.
The Arrow C Data Interface is a tiny, stable C ABI: two structs, ArrowSchema and ArrowArray, that describe a type and point to buffers, plus a release callback so the producer frees memory when the consumer is done. The C Stream Interface (ArrowArrayStream) adds a pull-based stream of batches. Any library can implement them without depending on Arrow's C++ code, which is how Rust, Go, R, DuckDB and others share data in-process with zero copies.
In Python these structs travel inside PyCapsules through dunder methods:
__arrow_c_schema__()for types and schemas__arrow_c_array__()for an array or record batch__arrow_c_stream__()for a table or stream of batches
PyArrow both produces and consumes them. pa.table(obj), pa.array(obj) and pa.RecordBatchReader.from_stream(obj) accept any object exposing these methods: a Polars DataFrame, a DuckDB relation, a pandas DataFrame with Arrow-backed columns, or your own class. Polars and others in turn accept PyArrow objects the same way. This protocol replaces older routes such as the DataFrame interchange protocol, which Polars 2 dropped.
To make your own object interoperable, implement __arrow_c_stream__(self, requested_schema=None) and return a capsule from a PyArrow reader you build internally.
Example
import pyarrow as pa
class Sensor:
def __init__(self):
self.t = pa.table({"ts": [1, 2, 3], "v": [0.5, 0.7, 0.2]})
def __arrow_c_stream__(self, requested_schema=None):
# Export our data through the C Stream Interface
return self.t.__arrow_c_stream__(requested_schema)
imported = pa.table(Sensor()) # any consumer can read it
print(imported.num_rows, imported.column_names) # 3 ['ts', 'v']
reader = pa.RecordBatchReader.from_stream(Sensor())
print(reader.schema.names, reader.read_all().num_rows) # ['ts', 'v'] 3
capsule = pa.int64().__arrow_c_schema__()
print(type(capsule).__name__) # PyCapsule
print(pa.DataType._import_from_c_capsule(capsule)) # int64Remember
- C Data Interface = stable ABI for schemas and arrays with a release callback
- Python exchanges it via
__arrow_c_schema__/array__/stream__capsules - Implement
__arrow_c_stream__to make your objects readable by every Arrow tool
Watch out for
- Relying on the deprecated DataFrame interchange protocol for new code instead of PyCapsules.