TopolearnFor imbalanced or cost-sensitive problems, these metrics tell you what accuracy hides.
For a positive class (fraud, disease, churn):
- Precision = TP / (TP + FP): of the cases flagged, how many were right.
- Recall = TP / (TP + FN): of the real positives, how many were caught.
- F1 is their harmonic mean, high only when both are.
classification_report(y_true, y_pred) prints all three per class plus macro and weighted averages. For multi-class, choose average="macro" (every class counts equally) or "weighted" (by support).
These depend on the decision threshold. predict labels a sample positive when its probability exceeds 0.5 (for most classifiers). Lowering the threshold raises recall and usually lowers precision. To evaluate the model independently of a threshold, use scores:
- ROC AUC (
roc_auc_score(y_true, proba)) is the probability that a random positive gets a higher score than a random negative; 0.5 is random guessing. - Average precision (
average_precision_score) summarises the precision-recall curve and is more informative than ROC AUC when positives are rare.
Pass probabilities or decision_function outputs, not hard labels, to these. precision_recall_curve and roc_curve give the full curves, and PrecisionRecallDisplay / RocCurveDisplay plot them.
Example
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score, recall_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, weights=[0.95], random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
clf = LogisticRegression().fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1] # score for the positive class
print(round(roc_auc_score(y_te, proba), 3)) # 0.905
print(round(average_precision_score(y_te, proba), 3)) # 0.793
print(round(recall_score(y_te, clf.predict(X_te)), 2)) # 0.67 (threshold 0.5)
print(round(recall_score(y_te, proba > 0.2), 2)) # 0.85 (lower threshold)Remember
- Precision: how many flagged are right; recall: how many positives are caught
- Threshold metrics need labels; ROC AUC and average precision need scores
- Prefer average precision when positives are rare
Watch out for
- Passing
predictoutput to roc_auc_score, which collapses the curve to one point.