TopolearnEarly stopping chooses the number of trees and prevents overfitting with almost no extra effort.
Pass validation data and a callback to fit:
model.fit(X_tr, y_tr, eval_X=X_val, eval_y=y_val, eval_metric="auc",
callbacks=[lgb.early_stopping(50), lgb.log_evaluation(100)])lgb.early_stopping(stopping_rounds)stops when a validation metric hasn't improved for that many rounds. By default it watches every metric and stops on whichever stalls first;first_metric_only=Truerestricts it to the first.lgb.log_evaluation(period)prints progress everyperiodrounds (0 for silence).lgb.record_evaluation(d)stores the full history in a dict; with the scikit-learn API,model.evals_result_holds it too.
After training, model.best_iteration_ gives the best round, and predict uses it automatically. The objective's default metric is always evaluated; eval_metric adds more.
Since 4.7, validation data is passed as eval_X and eval_y (tuples for several sets); the older eval_set=[(X, y)] still works but is deprecated. Early stopping considers every validation set, so don't include the training data there.
Set n_estimators generously with a small learning rate and let early stopping decide; keep the validation set separate from the test set.
Example
import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=6000, n_features=20, n_informative=8,
flip_y=0.05, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, random_state=0)
model = lgb.LGBMClassifier(n_estimators=5000, learning_rate=0.03, verbose=-1)
model.fit(X_tr, y_tr, eval_X=X_val, eval_y=y_val, eval_metric="auc",
callbacks=[lgb.early_stopping(100, verbose=False), lgb.log_evaluation(0)])
hist = model.evals_result_["valid_0"]
print(model.best_iteration_, sorted(hist)) # 80 ['auc', 'binary_logloss']
print(round(model.best_score_["valid_0"]["auc"], 4)) # 0.9637
logloss = hist["binary_logloss"]
print(logloss.index(min(logloss)) + 1) # 178: logloss was still improving!
# AUC stalled first and stopped training; use first_metric_only=True with the
# metric you care about listed first if that mattersRemember
- Early stopping is a callback:
callbacks=[lgb.early_stopping(n)] - Pass validation data as
eval_X/eval_y;predictusesbest_iteration_ - With several metrics, the first to stall stops training
Watch out for
- Adding eval_metric="auc" and not noticing that AUC, not logloss, decided when to stop.