TopolearnUsing a test whose assumptions don't fit the data produces confident but wrong conclusions.
The right test depends on the question and the data type.
Comparing groups of numbers:
- two independent groups:
ttest_ind(means, Welch), ormannwhitneyuwhen data are skewed, ordinal or have outliers, - paired data:
ttest_relorwilcoxon, - three or more groups:
f_oneway(ANOVA) orkruskal; follow up withtukey_hsdfor pairwise differences.
Categorical counts: chi2_contingency(table) tests independence in a contingency table and returns expected counts; for small 2x2 tables use fisher_exact. binomtest(k, n, p) checks a single proportion.
Relationships: pearsonr, spearmanr, kendalltau.
Distribution checks: shapiro and normaltest test normality, ks_2samp compares two whole distributions. Don't over-rely on normality tests: with large samples they reject trivial deviations, and with small ones they have little power. A histogram or Q-Q plot (stats.probplot) often says more.
Many tests at once inflate false positives: at alpha 0.05, 20 independent null tests produce one "significant" result on average. stats.false_discovery_control(pvalues) returns Benjamini-Hochberg adjusted p-values to compare with your threshold.
Example
import numpy as np
from scipy import stats
rng = np.random.default_rng(1)
a = rng.lognormal(3.0, 0.8, size=30) # skewed data, e.g. order values
b = rng.lognormal(3.6, 0.8, size=30)
print(round(stats.mannwhitneyu(a, b).pvalue, 4)) # 0.0025
table = np.array([[120, 80], # clicked / not, version A
[150, 50]]) # clicked / not, version B
chi2, p, dof, expected = stats.chi2_contingency(table)
print(round(p, 4), dof) # 0.002 1
g1, g2, g3 = rng.normal(5, 1, 20), rng.normal(5, 1, 20), rng.normal(6, 1, 20)
print(round(stats.f_oneway(g1, g2, g3).pvalue, 5)) # 3e-05
pvals = [0.001, 0.01, 0.03, 0.04, 0.2]
print(stats.false_discovery_control(pvals).round(3)) # [0.005 0.025 0.05 0.05 0.2 ]Remember
- Match the test to the data: means, ranks, counts or correlations
- Non-parametric tests (Mann-Whitney, Wilcoxon, Kruskal) suit skewed or ordinal data
- Adjust p-values when you run many tests
Watch out for
- Using a t-test on a 2x2 table of counts instead of a chi-squared or Fisher test.