Multi-objective optimization

Real models trade quality against cost, size or fairness; a single score hides those trade-offs.

Pass directions instead of direction and return a tuple:

study = optuna.create_study(directions=["maximize", "minimize"])
def objective(trial):
    ...
    return accuracy, latency_ms

There is no single best trial anymore. study.best_trials returns the Pareto front: trials for which no other trial is at least as good on every objective and strictly better on one. Choosing among them is a business decision (e.g. the most accurate model under 20 ms).

Samplers: in Optuna 5 the default is TPE, which supports multiple objectives; NSGAIISampler (genetic) is a classic choice for larger budgets, NSGAIIISampler for many objectives, and GPSampler for small budgets.

Things that change:

trial.values (plural) holds the objective tuple; study.best_value and best_params raise errors.
Pruning is not supported for multi-objective studies.
optuna.visualization.plot_pareto_front(study, target_names=[...]) shows the trade-off curve.
study.set_metric_names(["accuracy", "latency_ms"]) labels objectives in plots and dataframes.

If one objective is really a hard limit (e.g. model must be under 50 MB), consider constraints instead: a constraints_func passed to the sampler marks trials as infeasible, which is simpler than exploring a full front.

Example

import optuna

optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    n_layers = trial.suggest_int("n_layers", 1, 8)
    width = trial.suggest_int("width", 16, 512, log=True)
    accuracy = 1 - 1 / (1 + n_layers * width / 200)    # bigger is better...
    latency_ms = 0.5 + n_layers * width / 100           # ...but slower
    return accuracy, latency_ms

study = optuna.create_study(directions=["maximize", "minimize"],
                            sampler=optuna.samplers.NSGAIISampler(seed=0))
study.set_metric_names(["accuracy", "latency_ms"])
study.optimize(objective, n_trials=100)

front = sorted(study.best_trials, key=lambda t: t.values[1])
print(len(front) > 1)                       # True: several trade-offs
fast_enough = [t for t in front if t.values[1] <= 10]
best = max(fast_enough, key=lambda t: t.values[0])
print(best.params, [round(v, 3) for v in best.values])

Remember

Use directions=[...] and return a tuple of values
study.best_trials is the Pareto front; pick from it with your own rule
No pruning in multi-objective studies; consider constraints for hard limits

Watch out for

Calling study.best_params on a multi-objective study, which raises an error.

Check yourself

What does study.best_trials contain in a two-objective study?

Official docs: optuna.readthedocs.io/en/stable/tutorial/20_recipes/002_multi_objective.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks