Uncertainty: standard errors and p-values

statsmodels · Essentials · 12 min

statsmodels' main strength is reporting how sure you can be about each coefficient, and these three numbers are how it does that.

Your table is a sample: a few hundred flats out of all the flats that exist. Collect a different sample and the fitted slope would come out a little different. So a coefficient is an estimate, a best guess from the data you happen to have, and the question is how far off it might be.

The standard error answers that. It's the typical amount an estimate would move around if you repeated the whole data collection many times. A slope of 2.5 with a standard error of 0.05 is pinned down tightly; with a standard error of 2 it is very vague. More rows and less scatter give smaller standard errors.

A confidence interval turns that into a range. The usual 95% interval is roughly the estimate plus or minus two standard errors, say [2.4, 2.6]. Loosely: the values inside are the ones consistent with your data. If you repeated the study many times, about 95% of the intervals computed this way would contain the true value.

Often you want to know whether a predictor matters at all. A hypothesis test starts by assuming the boring answer, the null hypothesis: "the true coefficient is zero; this predictor has no effect". It then asks how surprising your estimate would be if that were true.

The answer is the p-value: the probability of getting an estimate at least this far from zero purely by chance, if the true value were zero. A small p-value (below 0.05 is the common convention) means chance alone is an unlikely explanation, and the result is called statistically significant.

Two warnings. A small p-value does not mean the effect is large or important; with lots of data, tiny effects become significant. And a large p-value does not prove there is no effect; it may only mean your data are too few or too noisy to tell.

All of these numbers are only trustworthy if the model's assumptions roughly hold, which later lessons show how to check. Using a sample to draw conclusions like this is called statistical inference.

Example

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(0)
df = pd.DataFrame({"area": rng.uniform(40, 150, 100), "luck": rng.normal(size=100)})
df["price"] = 20 + 2.5 * df["area"] + rng.normal(0, 15, 100)   # luck plays no part

res = smf.ols("price ~ area + luck", data=df).fit()
print(res.params.round(2).to_dict())   # estimates
print(res.bse.round(2).to_dict())      # standard errors: how far off they might be
print(res.conf_int().loc["area"].round(2).tolist())   # 95% interval around 2.5
print(res.pvalues.round(3).to_dict())  # area tiny (real effect), luck large (no evidence)

Remember

  • A coefficient is an estimate from a sample; its standard error says how much it could vary
  • A 95% confidence interval is roughly the estimate plus or minus two standard errors
  • The p-value asks how surprising the estimate would be if the true effect were zero
  • Significant does not mean large, and not significant does not mean zero

Watch out for

  • Treating p = 0.06 as "no effect" and p = 0.04 as "proven"; they are nearly the same amount of evidence.

Check yourself

  1. A coefficient is 3.0 with a standard error of 0.5. Which 95% confidence interval is about right?
  2. A predictor's p-value is 0.40. What can you conclude?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: www.statsmodels.org/stable/generated/statsmodels.regression.linear_model.RegressionResults.html