TopolearnLooking at the shape of data comes before modelling or testing it.
NumPy already gives mean, median, standard deviation and percentiles. scipy.stats adds statistics NumPy lacks and convenient summaries.
stats.describe(x) returns count, min/max, mean, variance (with ddof=1, the sample variance), skewness and kurtosis in one result object.
Shape of the distribution:
stats.skew(x): asymmetry; positive means a long right tail (incomes, latencies).stats.kurtosis(x): tail heaviness, as excess kurtosis by default (0 for a normal).
Robust statistics resist outliers:
stats.iqr(x): interquartile range,stats.median_abs_deviation(x, scale="normal"): a robust standard deviation estimate,stats.trim_mean(x, 0.1): mean after cutting 10% from each end,stats.mode(x): most common value.
stats.zscore(x) standardises values. Correlation between two variables: stats.pearsonr(x, y) (linear), stats.spearmanr(x, y) (rank-based, monotonic relationships, robust to outliers). Both return the coefficient and a p-value.
Most functions take an axis argument like NumPy and a nan_policy ("propagate", "omit" or "raise") for missing values. Note that NumPy's np.std defaults to ddof=0 (population) while describe reports the sample variance.
Example
import numpy as np
from scipy import stats
latency = np.array([12, 15, 14, 13, 16, 15, 14, 250]) # one outlier (ms)
d = stats.describe(latency)
print(d.mean, round(d.variance, 1)) # 43.625 6955.1
print(round(stats.skew(latency), 2)) # 2.27 (long right tail)
print(np.median(latency), stats.trim_mean(latency, 0.125)) # 14.5 14.5
print(round(stats.median_abs_deviation(latency, scale="normal"), 2)) # 1.48
x = np.arange(10)
y = x ** 3 # monotonic, not linear
print(round(stats.pearsonr(x, y).statistic, 3)) # 0.908
print(round(stats.spearmanr(x, y).statistic, 3)) # 1.0Remember
describegives a full numeric summary in one call- Prefer median, IQR, MAD and trimmed means when outliers are present
- Spearman captures monotonic relationships; Pearson only linear ones
Watch out for
- Comparing NumPy
std(ddof=0) with a sample standard deviation (ddof=1) and wondering why they differ.