TopolearnFraud, churn and defect detection are all imbalanced, and accuracy hides models that never predict the rare class.
With 2% positives, a model predicting "never" is 98% accurate and useless. Fix the evaluation first: use AUC, PRAUC (precision-recall AUC), F1, or precision and recall at a chosen threshold, and set eval_metric accordingly so early stopping optimises what matters.
Training-side options:
auto_class_weights="Balanced"weights classes inversely to their frequency;"SqrtBalanced"is a gentler versionclass_weights={0: 1, 1: 20}orscale_pos_weight=20(binary) for explicit weightssample_weight/ Poolweightfor row-level costs
Reweighting shifts predicted probabilities upward for the rare class, so they no longer represent real frequencies. If you need calibrated probabilities (for expected-cost calculations), either don't reweight and tune the decision threshold instead, or recalibrate afterwards. Threshold tuning is often the simplest and best lever: predict probabilities on validation data and pick the threshold that maximises F1 or meets a required precision. catboost.utils.select_threshold helps.
For multiclass, MultiClass is chosen automatically; predict_proba returns one column per class in the order of model.classes_. MultiClassOneVsAll treats classes independently. TotalF1 with average=Macro (as eval_metric="TotalF1:average=Macro") weights small classes equally.
Example
import numpy as np
from catboost import CatBoostClassifier
from sklearn.datasets import make_classification
from sklearn.metrics import f1_score, recall_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=6000, weights=[0.97], flip_y=0.01, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, stratify=y, random_state=0)
print(round(y.mean(), 3)) # about 0.03 positives
common = dict(iterations=400, verbose=0, random_seed=0, allow_writing_files=False,
eval_metric="PRAUC")
plain = CatBoostClassifier(**common).fit(X_tr, y_tr)
balanced = CatBoostClassifier(auto_class_weights="Balanced", **common).fit(X_tr, y_tr)
for name, m in [("plain", plain), ("balanced", balanced)]:
pred = m.predict(X_val)
print(name, round(recall_score(y_val, pred), 2), round(f1_score(y_val, pred), 2))
# balanced: higher recall, usually lower precision
proba = plain.predict_proba(X_val)[:, 1]
best_t = max(np.arange(0.05, 0.95, 0.05), key=lambda t: f1_score(y_val, proba >= t))
print(round(best_t, 2), round(f1_score(y_val, proba >= best_t), 2)) # tuned thresholdRemember
- Evaluate with PRAUC, AUC, F1 or precision/recall, not accuracy
- auto_class_weights, class_weights and scale_pos_weight reweight classes
- Reweighting distorts probabilities; threshold tuning keeps them calibrated
- predict_proba columns follow model.classes_ for multiclass
Watch out for
- Using the default 0.5 threshold on an imbalanced problem without checking precision and recall.