TopolearnNearly every real dataset has gaps, and how you handle them changes your results.
pandas uses several markers for "missing": NaN in float columns, None or NaN in object columns, NaT in datetime columns, and pd.NA in the nullable dtypes. You don't need to tell them apart in everyday code: `isna()` and `notna()` detect all of them. Never test with == np.nan, which is always False.
Finding gaps: df.isna().sum() per column, df[df["x"].isna()] for the rows.
Dropping with dropna():
df.dropna()drops any row containing a missing value, often far too aggressivesubset=["price"]considers only certain columnshow="all"drops only fully empty rows;thresh=3keeps rows with at least 3 values
Filling with fillna():
- a constant:
df["qty"].fillna(0) - per column:
df.fillna({"qty": 0, "note": ""}) - a statistic:
df["temp"].fillna(df["temp"].median()) - carry values forward or back in ordered data:
df["price"].ffill(),bfill()(the oldfillna(method="ffill")is deprecated) - interpolate numeric series:
s.interpolate()
A type gotcha: classic integer columns can't hold NaN, so one missing value turns an int64 column into float64 and a bool column into object. The nullable dtypes ("Int64", "boolean") avoid that by using pd.NA.
Before filling, ask why data is missing. Filling with 0 or the mean is a modelling decision: zero sales and unknown sales are different facts. Often it's worth adding an indicator column (df["price_missing"] = df["price"].isna()) before filling.
Example
import numpy as np
import pandas as pd
df = pd.DataFrame({
"day": pd.to_datetime(["2024-05-01", "2024-05-02", None, "2024-05-04"]),
"price": [10.0, np.nan, np.nan, 13.0],
"qty": [3, None, 2, 1],
"note": ["ok", None, "late", None],
})
print(df.isna().sum().to_dict()) # {'day': 1, 'price': 2, 'qty': 1, 'note': 2}
print(df["qty"].dtype) # float64 (int + missing)
print(len(df.dropna())) # 1 too aggressive
print(len(df.dropna(subset=["price"]))) # 2
filled = df.assign(
price=df["price"].ffill(), # [10, 10, 10, 13]
qty=df["qty"].fillna(0).astype(int),
note=df["note"].fillna(""),
)
print(filled["price"].tolist(), filled["qty"].tolist())
print(df["price"].interpolate().tolist()) # [10.0, 11.0, 12.0, 13.0]
print(pd.Series([1, None], dtype="Int64").tolist()) # [1, <NA>]Remember
isna()/notna()detect NaN, None, NaT and pd.NA alikedropna(subset=..., how=..., thresh=...)controls what gets droppedfillna,ffill,bfillandinterpolatefill gaps- Missing values turn int columns into float unless you use nullable dtypes
Watch out for
- Calling
df.dropna()on wide data and silently losing most rows because of one sparse column.