TopolearnYou need to know what drives predictions to debug models, catch leakage and explain results.
Tree ensembles expose feature_importances_, the impurity-based importance. It is cheap but computed on training data and biased towards features with many unique values, so a random ID column can look important.
Permutation importance is more reliable. permutation_importance(model, X_val, y_val, n_repeats=10, scoring=...) shuffles one column at a time on held-out data and measures how much the score drops. Works with any fitted estimator or pipeline, on raw input columns. Read importances_mean together with importances_std. Caveat: with strongly correlated features, shuffling one leaves its twin intact, so both can look unimportant; group or drop correlated features to check.
Partial dependence shows how predictions change as a feature varies, averaging over the others: PartialDependenceDisplay.from_estimator(model, X, ["age", ("age", "income")]). Setting kind="individual" draws ICE curves, one per sample, revealing heterogeneity the average hides.
For linear models, standardised coefficients are already interpretable. For individual predictions (why was this loan rejected?), the separate SHAP library is the common choice.
None of these tools measure causation: they describe what the model does, and the model learned correlations in the data.
Example
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
X, y = load_diabetes(return_X_y=True, as_frame=True)
X["random_id"] = np.random.default_rng(0).permutation(len(X)) # pure noise
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
rf = RandomForestRegressor(random_state=0).fit(X_tr, y_tr)
imp = dict(zip(X.columns, rf.feature_importances_.round(3)))
print(imp["bmi"], imp["random_id"]) # 0.353 0.055 (noise looks important!)
perm = permutation_importance(rf, X_te, y_te, n_repeats=10, random_state=0)
pi = dict(zip(X.columns, perm.importances_mean.round(3)))
print(pi["bmi"], pi["random_id"]) # 0.143 0.001 (noise correctly ~0)Remember
- Impurity importances are biased towards high-cardinality features
- Permutation importance on held-out data works with any model
- Partial dependence and ICE show how a feature shapes predictions
Watch out for
- Concluding a feature is unimportant when a correlated twin carries the same information.