TopolearnThe loss defines what the model optimises; picking the right one matters more than most hyperparameters.
loss_function is what training minimises; metrics are what you watch. Common losses:
- classification:
Logloss(binary, default for the classifier),CrossEntropy(probabilistic labels),MultiClass(chosen automatically when there are more than two classes) - regression:
RMSE(default),MAE,Quantile:alpha=0.9for prediction intervals,MAPE,PoissonandTweedie:variance_power=1.5for counts and insurance-style targets,Huber:delta=1.0for robustness to outliers - ranking:
YetiRank,PairLogit,QueryRMSE(withgroup_idin the Pool)
Parameters for a loss go after a colon, as above.
eval_metric selects the metric used for early stopping and choosing the best iteration, for example AUC or F1 while still optimising Logloss. custom_metric=["AUC", "Precision", "Recall"] logs extra metrics without affecting training. After fitting, model.get_best_score() and model.get_evals_result() return the values, and model.eval_metrics(pool, ["AUC"]) computes metrics per iteration on any data.
For classifiers, predict returns labels using a 0.5 probability threshold (binary). If the costs of errors are unequal, predict probabilities and choose your own threshold on validation data rather than relying on the default.
Example
import numpy as np
from catboost import CatBoostClassifier, CatBoostRegressor, Pool
from sklearn.datasets import make_classification, make_regression
X, y = make_classification(n_samples=600, n_classes=3, n_informative=5, random_state=0)
clf = CatBoostClassifier(iterations=200, verbose=0, allow_writing_files=False,
custom_metric=["Accuracy"])
clf.fit(X[:500], y[:500], eval_set=(X[500:], y[500:]))
print(clf.get_all_params()["loss_function"]) # MultiClass (auto)
print(sorted(clf.get_best_score()["validation"])) # ['Accuracy', 'MultiClass']
Xr, yr = make_regression(n_samples=500, noise=20, random_state=0)
q90 = CatBoostRegressor(loss_function="Quantile:alpha=0.9", iterations=300,
verbose=0, allow_writing_files=False).fit(Xr, yr)
print(round(np.mean(yr <= q90.predict(Xr)), 2)) # about 0.9 below the 90% quantile
f1 = clf.eval_metrics(Pool(X[500:], y[500:]), ["TotalF1"]) # per-iteration values
print(round(f1["TotalF1"][-1], 2)) # F1 after the last treeRemember
- loss_function is optimised; eval_metric picks the best iteration
- Losses take parameters after a colon, e.g. Quantile:alpha=0.9
- custom_metric logs extra metrics without changing training
- Choose decision thresholds yourself when error costs differ
Watch out for
- Optimising RMSE when the business cares about relative error or quantiles; pick the loss that matches the goal.