Filtering, sorting, grouping and joining tables

For simple pipelines you can stay in Arrow from file to file, skipping pandas entirely.

Tables have relational methods built on the compute kernels and the Acero engine:

t.filter(pc.field("amount") > 0) keeps rows matching an expression. pc.field("x") refers to a column; combine with &, |, ~ and methods like .isin([...]) or .is_null().
t.sort_by([("shop", "ascending"), ("amount", "descending")]).
t.group_by("shop").aggregate([("amount", "sum"), ("amount", "count"), ("id", "count_distinct")]) returns one row per group, with columns named like amount_sum. Group by several keys with a list.
left.join(right, keys="uid", join_type="left outer"). Join types are "inner", "left outer", "right outer", "full outer", "left semi", "left anti" and their right-side versions. Use right_keys= when names differ.

Aggregation names follow the compute functions: sum, mean, min, max, count, count_distinct, list (collect values), first, last, stddev.

The output order of group_by and join isn't guaranteed (pass use_threads=False to group_by for stable order), so sort the result if order matters.

This API is intentionally small. For complex analytics you would hand the Table to Polars, DuckDB or pandas, which is zero-copy or cheap, but for filtering, light aggregation and joining it saves a conversion.

Example

import pyarrow as pa
import pyarrow.compute as pc

orders = pa.table({"uid": [1, 1, 2, 3, 3], "amount": [10, -3, 7, 4, 6]})
users = pa.table({"uid": [1, 2, 4], "name": ["Ana", "Ben", "Di"]})

positive = orders.filter(pc.field("amount") > 0)
per_user = (
    positive.group_by("uid")
    .aggregate([("amount", "sum"), ("amount", "count")])
    .sort_by("uid")
)
print(per_user.to_pylist())
# [{'uid': 1, 'amount_sum': 10, 'amount_count': 1},
#  {'uid': 2, 'amount_sum': 7, 'amount_count': 1},
#  {'uid': 3, 'amount_sum': 10, 'amount_count': 2}]

joined = users.join(per_user, keys="uid", join_type="left outer").sort_by("uid")
print(joined.select(["name", "amount_sum"]).to_pydict())
# {'name': ['Ana', 'Ben', 'Di'], 'amount_sum': [10, 7, None]}

Remember

Filter with pc.field(...) expressions
group_by().aggregate([(col, func)]) names outputs col_func
Sort after group_by or join if order matters

Watch out for

Expecting group_by output in first-seen order with multithreading on.

Check yourself

What column name does t.group_by('k').aggregate([('v', 'mean')]) produce for the aggregate?

Official docs: arrow.apache.org/docs/python/compute.html#grouped-aggregations

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks