Validation sets and early stopping

Early stopping picks the number of trees automatically and is the main defence against overfitting.

Boosting keeps reducing training loss as trees are added, but validation loss eventually turns up. Early stopping watches a validation set and stops when its metric has not improved for a number of rounds.

With the scikit-learn API, set it in the constructor and pass validation data to fit:

model = XGBClassifier(n_estimators=5000, learning_rate=0.05,
                      early_stopping_rounds=50, eval_metric="logloss")
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)

After fitting, model.best_iteration and model.best_score describe the best round, and predict automatically uses the trees up to the best iteration. model.evals_result() holds the full metric history for plotting learning curves. If eval_set contains several sets, the last one is used for stopping.

Rules for doing it right:

The validation set must be separate from the test set; it is part of model selection.
Set n_estimators high and let early stopping cut it.
Inside cross-validation or a hyperparameter search, each fold needs its own validation split, or use xgb.cv, which applies early stopping across folds.
For the final model, either keep the early-stopped model or refit on all data with n_estimators set to around the best iteration (scaled up slightly for the extra data).

Example

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier

X, y = make_classification(n_samples=5000, n_features=20, flip_y=0.1, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, random_state=0)

model = XGBClassifier(n_estimators=5000, learning_rate=0.05, max_depth=4,
                      early_stopping_rounds=50, eval_metric="logloss")
model.fit(X_tr, y_tr, eval_set=[(X_tr, y_tr), (X_val, y_val)], verbose=False)

print(model.best_iteration)                   # 81 (instead of 5000)
hist = model.evals_result()
print(round(hist["validation_0"]["logloss"][-1], 3),   # 0.165 train keeps falling
      round(hist["validation_1"]["logloss"][-1], 3))   # 0.235 validation drifted up
print(round(model.best_score, 3))             # 0.23

Remember

Set early_stopping_rounds and eval_metric in the constructor, pass eval_set to fit
predict uses the best iteration automatically
The validation set is for model selection, not the final test

Watch out for

Using the test set as eval_set, which makes the reported test score optimistic.

Check yourself

With eval_set=[(X_tr, y_tr), (X_val, y_val)], which set does early stopping monitor?

Official docs: xgboost.readthedocs.io/en/stable/python/python_intro.html#early-stopping

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks