TopolearnDecisions based on probabilities, such as expected cost or risk tiers, need probabilities that mean what they say.
A classifier is calibrated if, among samples it gives probability 0.8, about 80% are positive. Good ranking (high AUC) does not imply calibration. Typical behaviour:
LogisticRegressionis usually well calibrated, unless strongly regularised.- Random forests push probabilities away from 0 and 1, boosting and SVMs can be over- or under-confident, and
GaussianNBis often extreme. class_weight="balanced"or resampling deliberately distorts probabilities.
Diagnose with a reliability diagram: calibration_curve(y_true, proba, n_bins=10) returns the observed fraction of positives per bin of predicted probability; CalibrationDisplay plots it. A summary number is brier_score_loss (mean squared error of probabilities, lower is better) or log_loss.
Fix with CalibratedClassifierCV(model, method=..., cv=5), which fits a calibrator on out-of-fold predictions:
method="sigmoid"(Platt scaling) fits a logistic curve; robust with little data.method="isotonic"fits a monotone step function; more flexible but needs more data (roughly 1,000+ samples) to avoid overfitting.
To calibrate an already-fitted model on a separate calibration set, wrap it as CalibratedClassifierCV(FrozenEstimator(model)). The calibration map is monotonic, so it preserves a model's ranking; with cv=5 the averaged fold models can still shift AUC slightly.
Example
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
X, y = make_classification(n_samples=20000, n_features=20, n_informative=4,
n_redundant=10, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
raw = GaussianNB().fit(X_tr, y_tr)
cal = CalibratedClassifierCV(GaussianNB(), method="isotonic", cv=5).fit(X_tr, y_tr)
for name, m in [("raw", raw), ("calibrated", cal)]:
p = m.predict_proba(X_te)[:, 1]
print(name, round(brier_score_loss(y_te, p), 4), round(roc_auc_score(y_te, p), 3))
# raw 0.1832 0.821
# calibrated 0.1566 0.829Remember
- High AUC does not mean calibrated probabilities
- Diagnose with calibration_curve and the Brier score
- Sigmoid for small data, isotonic for large; ranking is preserved
Watch out for
- Calibrating on the same data the model was trained on.