TopolearnReal data has gaps; you need to know how they propagate and how to skip them correctly.
NumPy has no dedicated missing-value type for every dtype. The common convention for floats is NaN: any arithmetic with NaN produces NaN, so a single missing value turns a.mean() into nan. That propagation is a feature; it stops gaps from silently skewing results.
To skip missing values deliberately:
- Use the nan-functions:
np.nansum,np.nanmean,np.nanstd,np.nanmax,np.nanargmax,np.nanpercentile. Note thatnanmeanof an all-NaN slice returns NaN with aRuntimeWarning, andnansumreturns 0. - Filter first:
a[~np.isnan(a)]. - Fill:
np.nan_to_num(a, nan=0.0)ornp.where(np.isnan(a), fill, a); for 1-D series,np.interpcan interpolate across gaps.
Integer and boolean arrays cannot hold NaN. Putting NaN into an integer array raises, and np.array([1, None]) silently becomes an object array, which is slow. Convert to float or keep a separate mask.
That separate mask is what masked arrays formalise. np.ma.masked_array(data, mask=...) pairs values with a boolean mask, and operations ignore masked entries: m.mean() averages only valid values, and results stay masked where inputs were masked. Helpers include np.ma.masked_invalid (mask NaN/inf), np.ma.masked_where(cond, a), m.filled(value) to get a plain array back, and m.compressed() for the valid values. They work for any dtype, including integers, and keep the original value under the mask.
Masked arrays are slower and less widely supported than plain arrays; many libraries just call np.asarray and drop the mask. In tabular work pandas' nullable dtypes are usually the better answer.
Example
import numpy as np
t = np.array([21.5, np.nan, 23.0, 22.1, np.nan])
print(t.mean(), np.nanmean(t)) # nan 22.2
print(np.isnan(t).sum()) # 2
print(t[~np.isnan(t)]) # [21.5 23. 22.1]
days = np.arange(t.size)
ok = ~np.isnan(t)
print(np.interp(days, days[ok], t[ok])) # gaps linearly filled
counts = np.array([5, -1, 7, -1, 3]) # -1 = sensor offline
m = np.ma.masked_equal(counts, -1)
print(m) # [5 -- 7 -- 3]
print(m.mean(), m.sum()) # 5.0 15
print(m.filled(0)) # [5 0 7 0 3]
bad = np.ma.masked_invalid(np.array([1.0, np.inf, np.nan, 4.0]))
print(bad.compressed()) # [1. 4.]Remember
- NaN propagates through arithmetic; use
nan*functions to skip it - Integer arrays can't hold NaN; mixing in None makes slow object arrays
np.mapairs data with a mask and works for any dtypefilled()andcompressed()convert masked arrays back to plain ones
Watch out for
- Testing for missing values with
a == np.nan, which is always False.