TopolearnTables are what you read from files and hand to other libraries, and the schema is the contract between them.
A Table is a set of named columns of equal length; each column is a ChunkedArray. A Schema lists the column names and types, as pa.field objects that can also say whether nulls are allowed and carry metadata.
Build a table with pa.table({"id": [1, 2], "name": ["a", "b"]}), pa.Table.from_pydict(...), or row-wise with pa.Table.from_pylist([{"id": 1, ...}, ...]). Pass schema= to enforce types instead of inferring them. Convert back with to_pydict() or to_pylist().
Inspect with t.schema, t.num_rows, t.num_columns, t.column_names, t.nbytes. Access a column with t["name"] or t.column(0).
Tables are immutable; every operation returns a new table that shares the unchanged column buffers, so it's cheap:
t.select(["id"]),t.drop_columns(["x"]),t.rename_columns([...])t.append_column("score", pa.array([...])),t.set_column(i, name, arr)t.slice(offset, length),t.cast(new_schema)pa.concat_tables([t1, t2])stacks tables with the same schema
A RecordBatch is like a table but each column is a single contiguous array. Batches are the unit of streaming: readers produce them, IPC streams carry them, and t.to_batches() or pa.Table.from_batches(...) move between the two forms.
Example
import pyarrow as pa
schema = pa.schema([
pa.field("id", pa.int32(), nullable=False),
pa.field("city", pa.string()),
pa.field("temp", pa.float32()),
])
t = pa.Table.from_pylist(
[{"id": 1, "city": "Oslo", "temp": 4.5},
{"id": 2, "city": "Lima", "temp": None}],
schema=schema,
)
print(t.num_rows, t.column_names) # 2 ['id', 'city', 'temp']
print(t.schema.field("id")) # pyarrow.Field<id: int32 not null>
t2 = t.append_column("hot", pa.array([False, None]))
print(t2.select(["city", "hot"]).to_pylist())
# [{'city': 'Oslo', 'hot': False}, {'city': 'Lima', 'hot': None}]
both = pa.concat_tables([t, t])
print(both.num_rows, both["id"].num_chunks) # 4 2Remember
- Table = named ChunkedArrays; Schema = names, types, nullability
- Pass an explicit
schema=when types matter - Operations return new tables that share buffers
Watch out for
- Concatenating tables whose schemas differ slightly (int32 vs int64): cast first or it fails.