Search spaces with suggest

The search space determines what Optuna can find; log scales and conditionals make it efficient.

Optuna is define-by-run: the search space is whatever your objective asks for while it runs, so plain Python controls it.

The suggest API:

trial.suggest_float(name, low, high); add log=True for scale-like parameters such as learning rates and regularization strengths, or step=0.1 for a discretized grid.
trial.suggest_int(name, low, high), with step= or log=True (e.g. layer widths 16 to 1024).
trial.suggest_categorical(name, choices), where choices are None, bools, ints, floats or strings.

Bounds are inclusive. Each parameter name must keep the same distribution across trials; suggesting "lr" with different bounds in different trials triggers an error or a warning.

Conditional spaces fall out naturally:

clf = trial.suggest_categorical("clf", ["svm", "rf"])
if clf == "svm":
    c = trial.suggest_float("svm_c", 1e-3, 1e3, log=True)
else:
    depth = trial.suggest_int("rf_depth", 2, 32, log=True)

Each trial records only the parameters it actually suggested. Loops work too, e.g. a variable number of layers each with their own units_{i} parameter.

Keep ranges realistic: an overly wide range wastes trials, while one that is too narrow may exclude the optimum. If the best value lands at a range edge, widen it.

Example

import optuna

optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    model = trial.suggest_categorical("model", ["linear", "mlp"])
    lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
    if model == "mlp":
        n_layers = trial.suggest_int("n_layers", 1, 3)
        widths = [trial.suggest_int(f"units_{i}", 16, 256, log=True)
                  for i in range(n_layers)]
        cost = abs(lr - 1e-3) * 100 + abs(sum(widths) - 200) / 100
    else:
        cost = abs(lr - 1e-2) * 100 + 1.0
    return cost

study = optuna.create_study(sampler=optuna.samplers.TPESampler(seed=1))
study.optimize(objective, n_trials=40)
t = study.trials[0]
print(sorted(t.params))       # only the params this trial suggested
print(study.best_params["model"])   # mlp

Remember

Suggest calls define the space as the objective runs
Use log=True for scale parameters like learning rates
Ordinary if-statements and loops create conditional spaces

Watch out for

Giving the same parameter name different ranges in different branches, which conflicts with the recorded distribution.

Check yourself

Which call is most appropriate for a learning rate between 1e-5 and 1e-1?

Official docs: optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks