TopolearnMost PyArrow work starts or ends in pandas or NumPy, and copying large data by accident is a common cause of slow code.
pa.Table.from_pandas(df) converts a pandas DataFrame into a Table; table.to_pandas() goes back. A non-default pandas index is stored as a column plus schema metadata, so it round-trips. Pass preserve_index=False to drop it.
Type mapping is mostly natural, with a few rules:
- Arrow integer columns with nulls become
float64in pandas by default (NaN for null), because NumPy ints have no null. Usetypes_mapper=pd.ArrowDtypeto get Arrow-backed pandas columns that keep exact types and nulls with no conversion. - Arrow strings become Python object or pandas string columns.
- Timestamps keep their unit and time zone.
For NumPy, pa.array(np_array) and arr.to_numpy() are zero-copy for numeric data without nulls. With nulls, or for strings, a copy is needed, so pass zero_copy_only=False. A zero-copy result is read-only, because Arrow memory is immutable.
pa.Table.from_pandas uses multiple threads, and conversions of large numeric columns are fast. Converting in a loop, row by row, is the slow path; always convert whole columns or tables. In pandas 2 and 3, pd.read_parquet(..., dtype_backend="pyarrow") and engine="pyarrow" use PyArrow underneath.
Example
import numpy as np
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({"n": [1, 2, 3], "s": ["x", "y", None]})
t = pa.Table.from_pandas(df, preserve_index=False)
print(t.schema.types) # [DataType(int64), DataType(large_string)] (pandas 3 str)
with_null = pa.table({"n": pa.array([1, None, 3])})
print(with_null.to_pandas()["n"].dtype) # float64
print(with_null.to_pandas(types_mapper=pd.ArrowDtype)["n"].dtype) # int64[pyarrow]
x = np.arange(5, dtype=np.int64)
arr = pa.array(x) # zero-copy from NumPy
back = arr.to_numpy() # zero-copy, read-only view
print(back.flags.writeable) # False
nulls = pa.array([1, None, 3])
try:
nulls.to_numpy() # zero_copy_only=True by default
except pa.ArrowInvalid:
print("nulls need a copy")
print(nulls.to_numpy(zero_copy_only=False)) # [ 1. nan 3.]Remember
- Integer columns with nulls become float64 in pandas unless you use ArrowDtype
- NumPy conversion is zero-copy only for numeric data without nulls
- Convert whole tables or columns, never row by row
Watch out for
- Writing into an array returned by zero-copy
to_numpy(); it is read-only.