TopolearnThis is the bread-and-butter use of Optuna and the template for tuning any model.
A realistic objective:
1. Suggests hyperparameters. 2. Builds the model with them. 3. Scores it on validation data, ideally with cross-validation (cross_val_score) for stability. 4. Returns the mean score.
def objective(trial):
params = {"n_estimators": trial.suggest_int("n_estimators", 50, 400),
"max_depth": trial.suggest_int("max_depth", 2, 16)}
return cross_val_score(RandomForestClassifier(**params), X, y, cv=5).mean()After optimization, retrain on the full training data with study.best_params and evaluate once on a held-out test set that the study never saw. Using the test set inside the objective leaks information and overstates performance.
Practical tips:
- Load data outside the objective (or pass it via a closure or
functools.partial) so it is not reloaded every trial. - Set model seeds for reproducibility; set
n_jobsin the model or in cross-validation, not both, to avoid oversubscribing CPUs. - Start with a modest budget (50 to 100 trials) and look at results before scaling up.
- For gradient boosting, tune learning rate, number of trees (or early stopping), depth or leaves, subsampling and regularization; most other knobs matter less.
Optuna does not care what the model is: the same pattern works for XGBoost, PyTorch or a whole data pipeline.
Example
import optuna
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, train_test_split
optuna.logging.set_verbosity(optuna.logging.WARNING)
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
def objective(trial):
params = {
"n_estimators": trial.suggest_int("n_estimators", 50, 300, step=50),
"max_depth": trial.suggest_int("max_depth", 2, 16, log=True),
"max_features": trial.suggest_categorical("max_features", ["sqrt", "log2", None]),
}
model = RandomForestClassifier(**params, random_state=0, n_jobs=-1)
return cross_val_score(model, X_tr, y_tr, cv=3).mean()
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=15)
final = RandomForestClassifier(**study.best_params, random_state=0).fit(X_tr, y_tr)
print(f"cv={study.best_value:.3f} test={final.score(X_te, y_te):.3f}") # both about 0.95Remember
- Score with cross-validation inside the objective
- Retrain with
best_paramsand test once on untouched data - Load data once outside the objective
Watch out for
- Evaluating on the test set inside the objective, which tunes to the test set and inflates the reported score.