TopolearnReal data has gaps, and mixing up null and NaN gives silently wrong counts and averages.
Polars has one missing-value marker for every type: null. It works the same in integer, string, date and boolean columns, so an integer column with a gap stays an integer column, unlike classic pandas where it turns into float.
NaN is different: it is a real floating-point value meaning "not a number", produced by things like 0.0 / 0.0. Polars does not treat NaN as missing. is_null() is false for NaN, null_count() doesn't count it, and mean() of a column containing NaN is NaN. Use is_nan() and fill_nan() for those, or convert with fill_nan(None).
Tools for nulls:
pl.col("x").is_null(),.is_not_null(),df.null_count()pl.col("x").fill_null(0),.fill_null(strategy="forward")or"mean"df.drop_nulls()ordf.drop_nulls(subset=["x"])pl.coalesce("a", "b", pl.lit(0))takes the first non-null value
Aggregations skip nulls: sum, mean, min and friends ignore them, and count() counts only non-null values while pl.len() counts rows. Comparisons with null return null, which filter treats as false.
When reading pandas data, pl.from_pandas(pdf, nan_to_null=True) (the default) converts NaN markers into proper nulls.
Example
import polars as pl
df = pl.DataFrame({"x": [1.0, None, float("nan"), 4.0]})
print(df.select(
nulls=pl.col("x").null_count(),
nans=pl.col("x").is_nan().sum(),
count=pl.col("x").count(), # non-null values, NaN included
rows=pl.len(),
).row(0)) # (1, 1, 3, 4)
print(df["x"].mean()) # nan (NaN poisons the mean)
clean = df.with_columns(pl.col("x").fill_nan(None))
print(clean["x"].mean()) # 2.5 (nulls are skipped)
print(clean["x"].fill_null(strategy="forward").to_list())
# [1.0, 1.0, 1.0, 4.0]Remember
- null is the missing marker for every dtype; NaN is a float value
- Aggregations skip nulls but not NaN
count()counts non-null values;pl.len()counts rows
Watch out for
- Using
is_null()to find NaN values in float columns: it returns false for NaN.