Metadata routing and sample weights

scikit-learn · Expert · 12 min

Without routing, weights silently stop at the first meta-estimator and your scores ignore them.

Some data belongs to samples but is not a feature: sample_weight, groups, exposure for insurance models. Historically, passing it through nested objects was awkward: fit(X, y, model__sample_weight=w) worked for pipelines, but cross_validate did not weight the scorer, and searches could not pass groups to inner splitters.

Metadata routing solves this explicitly. Enable it with sklearn.set_config(enable_metadata_routing=True), then each consumer declares what it wants:

  • est.set_fit_request(sample_weight=True) - use weights in fit,
  • est.set_score_request(sample_weight=True) - use them in score,
  • make_scorer(...).set_score_request(sample_weight=True) - weight the evaluation metric,
  • splitters such as GroupKFold request groups automatically.

Then pass the metadata once at the top: cross_validate(pipe, X, y, params={"sample_weight": w, "groups": g}, scoring=scorer, cv=GroupKFold()). Routing is strict: if metadata is passed and a consumer that could accept it has not said yes or no, scikit-learn raises an error instead of silently dropping it. Set sample_weight=False to explicitly decline.

Routing is opt-in and not yet supported by every meta-estimator; check get_metadata_routing() on an object to see what it routes. Third-party estimators participate by implementing the request methods (inherited from BaseEstimator).

Example

import numpy as np
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, make_scorer
from sklearn.model_selection import GroupKFold, cross_validate

sklearn.set_config(enable_metadata_routing=True)
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 3))
y = (X[:, 0] + rng.normal(0, 1, 200) > 0).astype(int)
w = rng.uniform(0.5, 2.0, size=200)
groups = np.repeat(np.arange(20), 10)

clf = LogisticRegression().set_fit_request(sample_weight=True)
scorer = make_scorer(accuracy_score).set_score_request(sample_weight=True)

res = cross_validate(clf, X, y, cv=GroupKFold(4), scoring=scorer,
                     params={"sample_weight": w, "groups": groups})
print(res["test_score"].round(3))   # weighted accuracy per fold
# [0.684 0.744 0.754 0.613]
sklearn.set_config(enable_metadata_routing=False)

Remember

  • Enable routing with set_config(enable_metadata_routing=True)
  • Consumers opt in with set_fit_request / set_score_request
  • Pass metadata once via params= at the top level

Watch out for

  • Weighting fit but not the scorer, so model selection ignores the weights.

Check yourself

  1. With metadata routing on, you pass sample_weight to cross_validate but never call set_fit_request on the classifier. What happens?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: scikit-learn.org/stable/metadata_routing.html

Metadata routing and sample weights · scikit-learn · Topolearn