TopolearnMany real statistics (medians, ratios, custom metrics) have no textbook test, and resampling handles them all.
Classical tests need a formula for the statistic's distribution. Resampling methods approximate it by simulation, so they work for medians, percentiles, ratios or any custom metric.
Bootstrap confidence intervals: stats.bootstrap((data,), statistic, confidence_level=0.95, n_resamples=9999, rng=...) resamples the data with replacement many times, recomputes the statistic, and builds an interval from the spread. The default method "BCa" corrects for bias and skew. The statistic should accept an axis argument (as NumPy functions do) so SciPy can vectorise; set vectorized=False otherwise. Pass several samples in the tuple and a statistic of several arguments for two-sample quantities, with paired=True for paired data.
Permutation tests: stats.permutation_test((a, b), statistic, ...) asks how often a statistic at least as extreme as the observed one arises when group labels are shuffled. That is exactly the null hypothesis "the groups are exchangeable", with no normality assumption. Use permutation_type="samples" for paired data and "pairings" for correlation.
Monte Carlo tests (stats.monte_carlo_test) generalise this to any null distribution you can sample from.
Caveats: the bootstrap needs a reasonably large sample (it cannot invent tails it never saw), and resampling assumes independent observations; clustered or time-series data need block methods.
Example
import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
a = rng.exponential(10, size=50) # e.g. session lengths, variant A
b = rng.exponential(14, size=50) # variant B
ci = stats.bootstrap((b,), np.median, confidence_level=0.95, rng=rng).confidence_interval
print(round(np.median(b), 2), round(ci.low, 2), round(ci.high, 2)) # 11.32 7.23 18.0
def diff_median(x, y, axis):
return np.median(x, axis=axis) - np.median(y, axis=axis)
res = stats.permutation_test((b, a), diff_median, n_resamples=9999,
alternative="greater", rng=rng)
print(round(res.statistic, 2), round(res.pvalue, 4)) # 3.36 0.101Remember
- Bootstrap gives confidence intervals for any statistic
- Permutation tests give p-values without distributional assumptions
- Write statistics that accept
axisso SciPy can vectorise
Watch out for
- Bootstrapping dependent data (time series, repeated measures) as if rows were independent.