TopolearnA/B tests and experiments are decided with these tests, and misreading them is very common.
A hypothesis test asks: if the null hypothesis were true (no difference, no effect), how surprising would data like ours be? The p-value is that probability of a result at least this extreme under the null.
The t-tests compare means:
stats.ttest_1samp(x, popmean): is the mean of x different from a value?stats.ttest_ind(a, b, equal_var=False): two independent groups.equal_var=Falsegives Welch's test, the safer default when variances may differ.stats.ttest_rel(before, after): paired measurements on the same units.
They return a result with statistic, pvalue, df, and a confidence_interval(confidence_level=0.95) method. Use alternative="greater" or "less" for one-sided tests decided before looking at data.
Reading results correctly:
- p < 0.05 means the data would be unusual under the null; it is not the probability that the null is true.
- A large p-value does not prove there is no effect; the sample may be too small.
- Statistical significance is not practical significance. Report the effect size and its confidence interval.
- Running many tests inflates false positives; correct with
stats.false_discovery_controlor a Bonferroni adjustment.
t-tests assume roughly normal means, which holds for moderate samples thanks to the central limit theorem.
Example
import numpy as np
from scipy import stats
rng = np.random.default_rng(42)
control = rng.normal(10.0, 2.0, size=40) # e.g. seconds on page
variant = rng.normal(11.0, 2.5, size=40)
res = stats.ttest_ind(variant, control, equal_var=False) # Welch's t-test
print(round(res.statistic, 2), round(res.pvalue, 4)) # 2.46 0.0162
ci = res.confidence_interval(0.95)
print(round(ci.low, 2), round(ci.high, 2)) # 0.18 1.73 (difference in means)
before = np.array([82, 75, 90, 68, 77])
after = np.array([79, 73, 85, 66, 75])
print(round(stats.ttest_rel(before, after).pvalue, 4)) # 0.0086Remember
- The p-value is P(data this extreme | null), not P(null | data)
- Use Welch (
equal_var=False) for two groups,ttest_relfor paired data - Report effect sizes and confidence intervals alongside p-values
Watch out for
- Testing many metrics and reporting only the ones with p < 0.05.