TopolearnMost of XGBoost's performance comes from a few parameters; knowing what each does makes tuning fast.
A small set of parameters matters most.
Learning rate and number of trees. learning_rate (eta, default 0.3) scales each tree's contribution. Smaller values need more trees (n_estimators) but usually generalise better. Typical final settings are 0.01-0.1 with hundreds to thousands of trees, chosen by early stopping.
Tree complexity.
max_depth(default 6): deeper trees capture more interactions but overfit faster; 3-8 is the usual range.min_child_weight(default 1): the minimum sum of instance weights (hessian) in a leaf; larger values make the model more conservative.gamma(min_split_loss, default 0): the minimum loss reduction needed to make a split.max_leaveswithgrow_policy="lossguide"grows trees leaf-wise, LightGBM style.
Randomness (stochastic boosting). subsample (rows per tree) and colsample_bytree (features per tree) around 0.6-0.9 reduce overfitting and speed up training. Set random_state for reproducibility.
Regularisation. reg_lambda (L2 on leaf weights, default 1) and reg_alpha (L1, default 0) shrink leaf values.
A practical order: fix a moderate learning rate (0.05-0.1) with early stopping, tune max_depth and min_child_weight, then the sampling parameters, then regularisation, and finally lower the learning rate for the final model.
Example
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from xgboost import XGBClassifier
X, y = make_classification(n_samples=1000, n_features=30, n_informative=8,
flip_y=0.1, random_state=0)
for params in [dict(max_depth=10, learning_rate=0.3), # very flexible
dict(max_depth=3, learning_rate=0.1),
dict(max_depth=3, learning_rate=0.1, subsample=0.8,
colsample_bytree=0.8, min_child_weight=5)]:
model = XGBClassifier(n_estimators=300, random_state=0, **params)
print(round(cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean(), 3), params)
# 0.932 {'max_depth': 10, 'learning_rate': 0.3}
# 0.929 {'max_depth': 3, 'learning_rate': 0.1}
# 0.925 {'max_depth': 3, 'learning_rate': 0.1, 'subsample': 0.8, ...}
# Close here: which setting wins is data-dependent, so measure with CVRemember
learning_rateandn_estimatorstrade off: lower rate, more treesmax_depth,min_child_weight,gammacontrol tree complexitysubsampleandcolsample_bytreeadd randomness that reduces overfitting
Watch out for
- Tuning n_estimators by grid search instead of using early stopping.