TopolearnKnowing TPE's mechanics explains its behaviour on small budgets, conditional spaces and parallel runs, and which options to tune.
Bayesian optimization usually models p(score | params). TPE instead models the parameters given the score:
1. After n_startup_trials random trials, sort completed trials by objective value and split them into a good group (the top fraction, controlled by gamma) and a bad group. 2. Fit two densities over parameter values: l(x) from good trials and g(x) from bad trials, using Parzen estimators (mixtures of kernels centred on observed values, plus a broad prior component). 3. Draw n_ei_candidates samples from l(x) and choose the one maximizing l(x) / g(x), which is proportional to expected improvement.
Consequences:
- Independent vs multivariate: classic TPE models each parameter separately and misses interactions. Optuna 5 enables multivariate TPE by default for single-objective studies, modelling parameters jointly.
group=Trueadditionally decomposes conditional spaces into subspaces that are modelled jointly. - Conditional spaces are natural because each parameter's density only uses trials where it was sampled.
- Startup: the first trials are random, so very small budgets gain little from TPE.
- Parallelism: with
constant_liar=Truerunning trials are treated as bad results, discouraging other workers from sampling the same spot. - Categoricals use weighted category frequencies rather than kernels.
TPE's cost per trial grows with the number of completed trials but stays small, which is why it scales to thousands of trials where Gaussian processes become slow.
Example
import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)
def objective(trial): # strong interaction between x and y
x = trial.suggest_float("x", -5, 5)
y = trial.suggest_float("y", -5, 5)
return (x - y) ** 2 + 0.1 * (x + y - 4) ** 2
def run(**kw):
vals = []
for seed in range(5):
s = optuna.create_study(sampler=optuna.samplers.TPESampler(seed=seed, **kw))
s.optimize(objective, n_trials=60)
vals.append(s.best_value)
return sum(vals) / len(vals)
print(f"independent : {run(multivariate=False):.3f}")
print(f"multivariate: {run(multivariate=True):.3f}") # usually lower (better)
print(f"short warmup: {run(multivariate=True, n_startup_trials=5):.3f}")Remember
- TPE splits trials into good and bad and maximizes the density ratio l(x)/g(x)
- Multivariate TPE (default for single objective in Optuna 5) captures parameter interactions
- constant_liar improves parallel runs; the first n_startup_trials are random
Watch out for
- Expecting TPE to beat random search with only 10 to 15 trials, when most of them are random startup trials.