Topolearn

Learn Optuna

Define-by-run hyperparameter optimization with smart sampling and pruning

15 lessons · 3 hours · 4 levels · covers Optuna 5.x

Optuna is a Python framework for hyperparameter optimization. You write an objective function that asks a trial for parameter values and returns a score; Optuna's samplers (TPE, CMA-ES, Gaussian processes and more) choose promising values, and pruners stop unpromising trials early. It scales from a notebook to many parallel workers sharing a database, with built-in visualization and multi-objective support.

Sign in free to track progressOfficial docs ↗

Before you start

Optuna builds on these; the tree links to the exact lessons you need.

PythonPractical level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics4 lessons · 44 min in total
2. PracticalUse it productively at work, with the usual idioms9 lessons · 2 hours in total
3. AdvancedThe less common features, performance and trade-offs12 lessons · 2 hours in total
4. ExpertInternals and the edge cases experts know15 lessons · 3 hours in total

The Optuna tree

OptunaObjective, trial and studyObjective, trial andstudySearch spaces with suggestSearch spaces withsuggestInspecting trials and resultsInspecting trials andresultsTuning a real modelTuning a real modelChoosing a samplerChoosing a samplerPruning unpromising trialsPruning unpromisingtrialsPersistent storage and resumingPersistent storageand resumingVisualizing studies and importanceVisualizing studiesand importanceBudgets, callbacks and enqueued trialsBudgets, callbacksand enqueued trialsDefining functionsDefining functionsBooleans, comparisons and ifBooleans,comparisons and if
Essentials and Practical shown (9 of 15 lessons); the full list is below. Click a lesson to read it.

All 15 lessons, in order

Studies, trials and search spaces

  1. Objective, trial and study · Essentials · 10 min

    Write an objective function, create a study, run optimize, and read the best parameters.

  2. Search spaces with suggest · Essentials · 12 min

    Define float, log-scale, integer, stepped and categorical parameters, including conditional spaces, at run time.

  3. Inspecting trials and results · Essentials · 10 min

    Explore every trial's params, values, states and timing, attach user attributes, and export to pandas.

  4. Tuning a real model · Essentials · 12 min

    Tune a scikit-learn model with cross-validation inside the objective, then retrain with the best parameters.

Samplers and analysis

  1. Choosing a sampler · Practical · 12 min

    Pick between TPE, random, CMA-ES, Gaussian process, grid and QMC samplers, and seed them for reproducibility.

  2. Visualizing studies and importance · Practical · 10 min

    Plot optimization history, parameter importances, slices and contours to understand the search.

Pruning and control

  1. Pruning unpromising trials · Practical · 14 min

    Report intermediate scores and let a pruner stop bad trials early with MedianPruner, Hyperband and friends.

  2. Budgets, callbacks and enqueued trials · Practical · 10 min

    Control how long a study runs, react to finished trials with callbacks, and seed the search with known-good parameters.

  3. Ask-and-tell and integrations · Advanced · 12 min

    Drive Optuna from your own loop with ask and tell, and use integrations for LightGBM, PyTorch Lightning and trackers.

Persistence and parallelism

  1. Persistent storage and resuming · Practical · 10 min

    Save studies to SQLite or a journal file, resume them later, and browse them with optuna-dashboard.

  2. Parallel and distributed optimization · Advanced · 12 min

    Run trials in parallel with threads, multiple processes, or many machines sharing a storage.

  3. Fault tolerance: heartbeats and retries · Expert · 12 min

    Keep long distributed studies healthy when workers crash or are preempted, using heartbeats, stale-trial handling and retries.

Multi-objective and internals

  1. Multi-objective optimization · Advanced · 12 min

    Optimize several objectives at once, such as accuracy and latency, and pick from the Pareto front.

  2. How TPE works · Expert · 14 min

    Understand the Tree-structured Parzen Estimator: good/bad splits, density ratios, multivariate modelling and constant liar.

  3. Custom samplers, pruners and constraints · Expert · 15 min

    Extend Optuna with your own sampler or pruner classes and handle constrained optimization.