TopolearnModern dtypes fix decades-old gotchas (ints turning into floats, text as slow objects) and match databases and Parquet.
Classic pandas columns are NumPy arrays, which have no missing marker for integers or booleans. pandas therefore upcasts: one missing value makes an int column float64 and a bool column object. Two families of dtypes fix that.
Nullable NumPy-backed dtypes pair a data array with a validity mask: "Int64", "Int32", "UInt8" (capitalised), "Float64", "boolean" and "string". Missing values are `pd.NA`, which follows three-valued logic like SQL's NULL: pd.NA == 1 is NA, True | pd.NA is True, False & pd.NA is False. Filtering with a boolean mask containing NA treats NA as False in loc/[].
Arrow-backed dtypes store data in Apache Arrow memory via PyArrow: pd.ArrowDtype(pa.int64()), written as strings like "int64[pyarrow]", "string[pyarrow]", "timestamp[us][pyarrow]", plus types NumPy lacks, such as lists, structs and decimals. Arrow strings are much faster and smaller than Python objects. Arrow is also the in-memory format of Parquet readers, Polars and DuckDB, so conversions can be zero-copy.
Getting them:
df.convert_dtypes()picks nullable types for existing columnspd.read_csv(..., dtype_backend="numpy_nullable")or"pyarrow"; same forread_parquet,read_sqlastype("Int64"),astype("string[pyarrow]")
String defaults changed in pandas 3.0: text columns get the str dtype, backed by PyArrow if installed. For compatibility it uses NaN (not pd.NA) as its missing value; the explicit "string" dtype keeps pd.NA semantics. In pandas 2.x text is object unless you ask otherwise.
Caveats: a few operations still fall back to slower paths or convert to NumPy, and passing nullable columns to libraries that expect plain NumPy may need to_numpy(dtype=float, na_value=np.nan).
Example
import pandas as pd
import pyarrow as pa
classic = pd.Series([1, None, 3])
print(classic.dtype) # float64 (upcast)
nullable = pd.Series([1, None, 3], dtype="Int64")
print(nullable.dtype, nullable.tolist()) # Int64 [1, <NA>, 3]
print((nullable > 1).tolist()) # [False, <NA>, True]
print(nullable[nullable > 1].tolist()) # [3] NA treated as False
print(pd.NA | True, pd.NA & False) # True False
df = pd.DataFrame({"n": [1, 2, None], "ok": [True, None, False], "s": ["a", None, "c"]})
print(df.convert_dtypes().dtypes.astype(str).tolist()) # ['Int64', 'boolean', 'string']
arrow = pd.Series(["x", "yy", None], dtype="string[pyarrow]")
print(arrow.str.len().tolist()) # [1, 2, <NA>]
lists = pd.Series([[1, 2], [3]], dtype=pd.ArrowDtype(pa.list_(pa.int64())))
print(lists.dtype) # list<item: int64>[pyarrow]
print(nullable.to_numpy(dtype="float64", na_value=float("nan"))) # [ 1. nan 3.]Remember
- Nullable dtypes (
Int64,boolean,string) keep types and usepd.NA pd.NAfollows three-valued logic like SQL NULL- Arrow-backed dtypes are fast, compact and interoperate with Parquet, Polars and DuckDB
- pandas 3.0 defaults text to a (PyArrow-backed)
strdtype
Watch out for
- Using
if value is np.nanchecks on nullable data, where missing values arepd.NA; usepd.isna(value).