TopolearnMany outcomes are not normal; GLMs model them correctly instead of forcing them into OLS.
A generalised linear model has three parts: a linear predictor (b0 + b1*x1 + ...), a link function connecting it to the mean of y, and a family describing y's distribution. In statsmodels: smf.glm("y ~ x", data=df, family=sm.families.Poisson()).fit().
Common choices:
Binomial()(logit link): binary outcomes or proportions,Poisson()(log link): counts; coefficients exponentiate to rate ratios,NegativeBinomial(alpha=...)(orsmf.negativebinomial): counts with more variance than the mean (overdispersion),Gamma(link=sm.families.links.Log()): positive, right-skewed amounts such as claim sizes,Tweedie(var_power=1.5): zero-inflated continuous data such as insurance losses.
For rates, include exposure: claims per policy-year or clicks per impression. Pass exposure=df["years"] (or offset=np.log(df["years"])) so the model estimates a rate, not a raw count.
Check overdispersion for Poisson models: res.pearson_chi2 / res.df_resid well above 1 means standard errors are too small. Switch to negative binomial, or use scale="X2" in fit to rescale them.
GLM results include deviance, aic, pearson_chi2, and predict returns values on the response scale (counts, probabilities) by default.
Example
import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf
rng = np.random.default_rng(6)
n = 1000
df = pd.DataFrame({"young": rng.integers(0, 2, n), "years": rng.uniform(0.5, 3, n)})
rate = np.exp(-2.0 + 0.6 * df["young"]) # claims per policy-year
df["claims"] = rng.poisson(rate * df["years"])
res = smf.glm("claims ~ young", data=df, family=sm.families.Poisson(),
exposure=df["years"]).fit()
print(res.params.round(2).to_dict()) # {'Intercept': -1.99, 'young': 0.58}
print(np.exp(res.params["young"]).round(2)) # 1.78 (young drivers: 78% higher claim rate)
print(round(res.pearson_chi2 / res.df_resid, 2)) # 0.89 (no overdispersion)Remember
- Pick the family from the outcome type and the link from how effects combine
- Use
exposure=oroffset=to model rates - Check Poisson overdispersion via Pearson chi² / df
Watch out for
- Modelling counts per varying exposure without an offset, which confuses rate with exposure.