TopolearnReal data includes JSON-like nesting, repeated labels and timestamps, and Arrow's types for them are shared by every Arrow-based tool.
Arrow has rich types beyond plain numbers and strings.
Nested types:
pa.list_(pa.int32()): a variable-length list per row.pa.list_(pa.float32(), 3)is fixed-size, good for vectors.pa.struct([("lat", pa.float64()), ("lon", pa.float64())]): named fields per row. Access fields withpc.struct_field(arr, "lat")orarr.field("lat");table.flatten()turns struct columns intoparent.childcolumns.pa.map_(pa.string(), pa.int64()): key/value pairs per row.
pc.list_value_length, pc.list_flatten and pc.list_parent_indices work on lists.
Dictionary encoding stores each distinct value once and the column as small integer indices: arr.dictionary_encode() or the type pa.dictionary(pa.int8(), pa.string()). It is how pandas Categorical and Polars Categorical map to Arrow, and it shrinks columns with repeated labels.
Temporal types: pa.date32() (days), pa.timestamp(unit, tz=None) with unit "s", "ms", "us" or "ns", pa.duration(unit) and pa.time64("us"). A timestamp with a tz stores UTC instants and displays in that zone; without one it is a naive wall-clock time. Casting between units can overflow or truncate; nanoseconds cover only about the years 1677 to 2262.
Example
import pyarrow as pa
import pyarrow.compute as pc
tags = pa.array([["a", "b"], [], None])
print(tags.type, pc.list_value_length(tags).to_pylist())
# list<item: string> [2, 0, None]
geo = pa.array([{"lat": 59.9, "lon": 10.7}, {"lat": -12.0, "lon": -77.0}])
t = pa.table({"geo": geo})
print(t.flatten().column_names) # ['geo.lat', 'geo.lon']
cities = pa.array(["Oslo", "Lima", "Oslo", "Oslo"])
enc = cities.dictionary_encode()
print(enc.dictionary.to_pylist(), enc.indices.to_pylist())
# ['Oslo', 'Lima'] [0, 1, 0, 0]
ts = pa.array([1_767_225_600], type=pa.timestamp("s", tz="Europe/Oslo"))
print(ts[0].as_py()) # 2026-01-01 01:00:00+01:00Remember
- list, fixed-size list, struct and map cover nested data
- Dictionary encoding = distinct values + integer indices
- Timestamps have a unit and an optional time zone
Watch out for
- Casting ns timestamps of far-past or far-future dates and hitting overflow.