TopolearnGood hyperparameters often make the difference between an average and an excellent boosted model.
Because XGBClassifier is a scikit-learn estimator, the usual model selection tools work: RandomizedSearchCV, GridSearchCV, HalvingRandomSearchCV, or a loop with an external optimiser such as Optuna.
Search spaces that work well:
max_depth: integers 3-10,min_child_weight: log-uniform 1-50,subsample,colsample_bytree: uniform 0.5-1.0,reg_lambda,reg_alpha: log-uniform 1e-3 to 10,gamma: 0-5,learning_rate: fix it at 0.05-0.1 during the search, lower it for the final fit.
Random search with 30-100 candidates usually beats a grid of the same size.
Number of trees: either include n_estimators in the search with a fixed learning rate, or use early stopping inside each fit. Early stopping inside GridSearchCV requires an eval_set, which must not be the CV fold being scored; a clean approach is to tune n_estimators explicitly, or to use xgb.cv per candidate in a custom loop.
XGBoost is already multi-threaded (n_jobs). Running a search with n_jobs=-1 on top oversubscribes the CPU; parallelise at one level only.
If preprocessing is needed (for example a TargetEncoder or ColumnTransformer), put it and the model in a Pipeline and search over xgbclassifier__max_depth and friends, so preprocessing is refitted in every fold.
Example
from scipy.stats import loguniform, randint, uniform
from sklearn.datasets import make_classification
from sklearn.model_selection import RandomizedSearchCV
from xgboost import XGBClassifier
X, y = make_classification(n_samples=2000, n_features=25, n_informative=10,
flip_y=0.05, random_state=0)
space = {"max_depth": randint(3, 9), "min_child_weight": loguniform(1, 50),
"subsample": uniform(0.5, 0.5), "colsample_bytree": uniform(0.5, 0.5),
"reg_lambda": loguniform(1e-3, 10), "n_estimators": randint(100, 600)}
search = RandomizedSearchCV(XGBClassifier(learning_rate=0.05, n_jobs=4),
space, n_iter=25, cv=4, scoring="roc_auc",
random_state=0, n_jobs=1) # parallelise in XGBoost only
search.fit(X, y)
print(round(search.best_score_, 4)) # 0.932
print({k: round(float(v), 2) for k, v in search.best_params_.items()})
# {'colsample_bytree': 0.96, 'max_depth': 6.0, 'min_child_weight': 1.04,
# 'n_estimators': 358.0, 'reg_lambda': 0.0, 'subsample': 0.86}Remember
- Random search over depth, child weight, sampling and regularisation works well
- Fix the learning rate while searching; lower it for the final model
- Parallelise either the search or XGBoost, not both
Watch out for
- Passing the CV validation fold as eval_set inside a search, leaking it into model selection.