Inspecting trials and results

Optuna · Essentials · 10 min

The best value is only part of the story; trial history shows sensitivity, failures and where to search next.

study.trials is a list of FrozenTrial objects. Each has number, params, value (or values for multi-objective), state, datetime_start, datetime_complete, duration, user_attrs and intermediate values reported for pruning.

States tell you what happened:

  • COMPLETE: returned a value.
  • PRUNED: stopped early by a pruner.
  • FAIL: the objective raised or returned NaN/None.
  • RUNNING and WAITING (queued with enqueue_trial).

Filter with study.get_trials(states=(optuna.trial.TrialState.COMPLETE,)).

study.trials_dataframe() returns a pandas DataFrame with columns like number, value, params_lr, state and duration, which is the quickest way to sort, group or plot results.

Store extra information on a trial with trial.set_user_attr("val_f1", f1), for example secondary metrics, the epoch with the best score, or a model path. These appear as user_attrs_* columns. Study-level notes go in study.set_user_attr.

If the objective raises an exception, the trial is marked FAIL and, by default, the exception propagates and stops optimize. Pass catch=(ValueError,) to record such failures and continue. Returning float("nan") also marks the trial as failed without stopping the study.

Example

import optuna

optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    x = trial.suggest_float("x", 0, 4)
    if x > 3.5:
        raise ValueError("unstable region")      # becomes a FAIL trial
    trial.set_user_attr("x_squared", x * x)
    return (x - 1) ** 2

study = optuna.create_study(sampler=optuna.samplers.RandomSampler(seed=0))
study.optimize(objective, n_trials=20, catch=(ValueError,))

df = study.trials_dataframe()
print(df["state"].value_counts().to_dict())   # e.g. {'COMPLETE': 18, 'FAIL': 2}
print(df.sort_values("value").iloc[0][["params_x", "user_attrs_x_squared"]].round(2).to_dict())

done = study.get_trials(states=(optuna.trial.TrialState.COMPLETE,))
print(all(t.value is not None for t in done))  # True

Remember

  • study.trials holds FrozenTrials with params, value, state and timing
  • trials_dataframe() exports everything to pandas
  • Use set_user_attr for extra metrics; catch= keeps the study going after errors

Watch out for

  • Letting one bad configuration's exception abort a long study instead of using catch= or returning NaN.

Check yourself

  1. An objective raises ZeroDivisionError for some parameters and you call optimize(..., catch=(ZeroDivisionError,)). What happens on that error?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: optuna.readthedocs.io/en/stable/tutorial/20_recipes/003_attributes.html