TopolearnCompute functions let you transform Arrow data directly, without converting to pandas or NumPy first.
import pyarrow.compute as pc exposes hundreds of vectorized kernels written in C++. They take arrays, chunked arrays or scalars and return new arrays.
- Arithmetic:
pc.add(a, b),pc.multiply,pc.divide,pc.round(x, 2). The*_checkedvariants such aspc.add_checkedraise on overflow instead of wrapping. - Comparisons and logic:
pc.greater(a, 3),pc.equal,pc.and_,pc.invert,pc.is_null,pc.is_in(a, value_set=pa.array([...])). - Strings:
pc.utf8_upper,pc.utf8_length,pc.match_substring(a, "abc"),pc.match_substring_regex,pc.replace_substring,pc.split_pattern. - Temporal:
pc.year,pc.day_of_week,pc.strftime,pc.strptime. - Aggregations return scalars:
pc.sum,pc.mean,pc.min_max,pc.count,pc.count_distinct,pc.value_counts. - Selection:
pc.filter(values, mask),pc.take(values, indices),pc.if_else(cond, a, b),pc.fill_null.
Nulls propagate: null + 1 is null, and aggregations skip nulls by default (skip_nulls=True), with min_count deciding when the result itself becomes null.
Many functions have an options object for extra settings, e.g. pc.round(x, ndigits=1) or pc.sort_indices(t, sort_keys=[("x", "descending")]). pc.list_functions() lists everything available.
Example
import pyarrow as pa
import pyarrow.compute as pc
price = pa.array([10.0, 25.5, None, 8.0])
qty = pa.array([2, 1, 4, 5])
names = pa.array(["Tea", "cake", "jam", "Tea"])
revenue = pc.multiply(price, qty)
print(revenue.to_pylist()) # [20.0, 25.5, None, 40.0]
print(pc.sum(revenue).as_py()) # 85.5 (null skipped)
mask = pc.and_(pc.greater(qty, 1), pc.is_valid(price))
print(pc.filter(names, mask).to_pylist()) # ['Tea', 'Tea']
print(pc.utf8_lower(names).to_pylist()) # ['tea', 'cake', 'jam', 'tea']
print(pc.value_counts(pc.utf8_lower(names)).to_pylist()[0])
# {'values': 'tea', 'counts': 2}
print(pc.if_else(pc.greater(qty, 3), "bulk", "single").to_pylist())
# ['single', 'single', 'bulk', 'bulk']Remember
pyarrow.compute= vectorized C++ kernels on Arrow data- Nulls propagate in arithmetic and are skipped in aggregations
- Use
*_checkedarithmetic when overflow must not wrap silently
Watch out for
- Looping over
arr.to_pylist()in Python for something a compute kernel does directly.