TopolearnFraud, churn and failure prediction are all imbalanced, and naive training optimises for the majority.
When positives are rare, a model can score high accuracy by ignoring them. Handle this in layers.
Measure the right thing. Use average precision, ROC AUC, recall at a fixed precision, or a business cost, with stratified cross-validation so each fold keeps some positives.
Weight the classes. Most classifiers accept class_weight="balanced", which weights samples inversely to class frequency, or a dict like {0: 1, 1: 10}. Estimators without that option usually accept sample_weight in fit. Weighting shifts predicted probabilities, so they are no longer calibrated.
Choose the threshold. Often the model ranks well (good AUC) and the real problem is the default 0.5 threshold. TunedThresholdClassifierCV(model, scoring="f1") finds, by cross-validation, the threshold that maximises a metric, and then predicts with it. For a fixed business rule use FixedThresholdClassifier(model, threshold=0.3). A custom scorer from make_scorer lets you optimise expected cost or profit directly.
Resampling (SMOTE, undersampling) lives in the separate imbalanced-learn library. It is often no better than class weights plus threshold tuning, and must be done inside the pipeline on training folds only, never before splitting.
Above all, get more positive examples or better features if you can; no reweighting creates signal that isn't there.
Example
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, recall_score
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
X, y = make_classification(n_samples=5000, weights=[0.97], class_sep=0.5,
flip_y=0, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
base = LogisticRegression().fit(X_tr, y_tr)
weighted = LogisticRegression(class_weight="balanced").fit(X_tr, y_tr)
tuned = TunedThresholdClassifierCV(LogisticRegression(), scoring="f1").fit(X_tr, y_tr)
for name, m in [("base", base), ("weighted", weighted), ("tuned", tuned)]:
pred = m.predict(X_te)
print(name, round(f1_score(y_te, pred), 3), round(recall_score(y_te, pred), 3))
# base 0.519 0.378 (misses most positives)
# weighted 0.274 0.892 (catches them, but many false alarms)
# tuned 0.61 0.676 (best F1)
print(round(tuned.best_threshold_, 3)) # 0.252Remember
- Evaluate imbalanced problems with ranking metrics or costs, not accuracy
class_weight="balanced"reweights training; it also distorts probabilitiesTunedThresholdClassifierCVpicks the threshold that optimises your metric
Watch out for
- Oversampling the whole dataset before splitting, which puts copies of test rows in training.