TopolearnExpensive trials only finish in reasonable time when many run at once.
Optuna parallelizes by having several workers share one study through storage. Each worker asks the storage for a new trial, the sampler reads completed trials, and results are written back.
Three levels:
- Threads:
study.optimize(objective, n_trials=100, n_jobs=4). Easy, but Python's GIL means it only helps when the objective releases the GIL (NumPy, scikit-learn with native code, GPU training, I/O). - Processes on one machine: start the same script several times, each calling
create_study(..., storage=..., load_if_exists=True)andoptimize. Use JournalStorage with a file backend, or a database. - Many machines: point all workers at PostgreSQL or MySQL (or a gRPC proxy server in front of the database for very many workers, via
run_grpc_proxy_serverandGrpcStorageProxy). Kubernetes jobs or a cluster scheduler simply launch N copies.
Caveats:
- Model-based samplers lose some efficiency when many trials run simultaneously, since they cannot learn from unfinished ones. TPE's
constant_liaroption (on by default in Optuna 5) treats running trials as if they had a poor result to avoid sampling the same region repeatedly. - Use
MaxTrialsCallbackto cap total trials across workers, because each worker'sn_trialscounts only its own. - Do not rely on seeds for reproducibility in parallel runs.
- Configure heartbeats so trials from crashed workers do not stay RUNNING forever.
Example
import optuna
from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend
from optuna.study import MaxTrialsCallback
from optuna.trial import TrialState
optuna.logging.set_verbosity(optuna.logging.WARNING)
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 1) ** 2
# Each worker process runs this same code (e.g. `python tune.py` started 4 times).
storage = JournalStorage(JournalFileBackend("parallel_journal.log"))
study = optuna.create_study(study_name="shared", storage=storage, load_if_exists=True)
study.optimize(
objective,
n_jobs=2, # threads inside this worker
callbacks=[MaxTrialsCallback(40, states=(TrialState.COMPLETE,))],
)
print(len(study.get_trials(states=(TrialState.COMPLETE,))) >= 40) # TrueRemember
- Workers coordinate only through shared storage
n_jobsuses threads (GIL-bound); separate processes or machines scale further- Cap total trials with MaxTrialsCallback; constant_liar reduces duplicate sampling
Watch out for
- Expecting n_jobs=8 to speed up a pure-Python objective; threads share the GIL, so use processes.