TopolearnChoosing the number of iterations on validation data is the single most important guard against overfitting in boosting.
More trees keep lowering the training loss forever, but the validation loss eventually bottoms out and rises: the model starts memorising noise. CatBoost's overfitting detector handles this.
eval_set=(X_val, y_val)or a validation Pool: data evaluated after every iterationearly_stopping_rounds=50: stop if the eval metric hasn't improved for 50 iterationsuse_best_model=True(the default when an eval set is given): keep only the trees up to the best iteration
After fitting, model.get_best_iteration() and model.get_best_score() report where it stopped, and model.tree_count_ shows the final size. plot=True in fit draws live curves in Jupyter.
Set iterations generously (several thousand) with a smaller learning_rate and let early stopping decide the actual count. Lower learning rates generally need more trees but generalise a little better.
The caveat: the validation set is now used to choose the model, so its score is optimistically biased. Keep a separate test set for the final, honest evaluation, or use cross-validation to choose the iteration count and then retrain on all the data with that fixed number.
For time series, the validation set must come after the training period, never a random split, or the stopping point will be chosen on leaked future information.
Example
from catboost import CatBoostClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=3000, n_features=20, flip_y=0.1, random_state=1)
X_tmp, X_test, y_tmp, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_tmp, y_tmp, test_size=0.25, random_state=0)
model = CatBoostClassifier(iterations=5000, learning_rate=0.05, eval_metric="AUC",
early_stopping_rounds=100, verbose=0, random_seed=0,
allow_writing_files=False)
model.fit(X_tr, y_tr, eval_set=(X_val, y_val), use_best_model=True)
print(model.get_best_iteration() < 5000) # True: stopped early
print(model.tree_count_ == model.get_best_iteration() + 1) # True: trimmed
print(round(model.get_best_score()["validation"]["AUC"], 3)) # best validation AUC
print(round(model.score(X_test, y_test), 3)) # honest accuracy on untouched test setRemember
- eval_set + early_stopping_rounds stop training when validation stops improving
- use_best_model keeps only trees up to the best iteration
- Set many iterations and a moderate learning rate; let early stopping decide
- Keep a separate test set; the validation score is optimistically biased
Watch out for
- Reporting the early-stopping validation score as the model's expected performance.