Topolearn

Learn XGBoost

Fast, regularised gradient-boosted trees for tabular data

15 lessons · 3 hours · 4 levels · covers XGBoost 3.x

XGBoost is an optimised gradient boosting library that builds ensembles of decision trees using second-order gradients and built-in regularisation. It handles missing values and categorical features natively, scales from a laptop to GPUs and clusters, and offers both a scikit-learn compatible API and a lower-level native API. It is one of the strongest and most widely used models for tabular prediction tasks.

Sign in free to track progressOfficial docs ↗

Before you start

XGBoost builds on these; the tree links to the exact lessons you need.

PythonEssentials levelscikit-learnEssentials level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics5 lessons · 48 min in total
2. PracticalUse it productively at work, with the usual idioms9 lessons · 2 hours in total
3. AdvancedThe less common features, performance and trade-offs12 lessons · 2 hours in total
4. ExpertInternals and the edge cases experts know15 lessons · 3 hours in total

The XGBoost tree

XGBoostXGBoost with the scikit-learn APIXGBoost with thescikit-learn APIObjectives and evaluation metricsObjectives andevaluation metricsCore hyperparametersCorehyperparametersMissing values and categorical featuresMissing values andcategorical featuresSaving and loading modelsSaving and loadingmodelsValidation sets and early stoppingValidation sets andearly stoppingThe native API: DMatrix, train and cvThe native API:DMatrix, train and cvTuning with scikit-learn toolsTuning withscikit-learn toolsFeature importance and SHAP valuesFeature importanceand SHAP valuesThe estimator API: fit, predict, transformThe estimator API: fit,predict, transformTrain/test splitTrain/test splitRegression metricsRegression metricsDecision treesDecision treesHyperparameter searchHyperparametersearchModel inspection and feature importanceModel inspection andfeature importance
Essentials and Practical shown (9 of 15 lessons); the full list is below. Click a lesson to read it.

All 15 lessons, in order

Getting started

  1. XGBoost with the scikit-learn API · Essentials · 9 min

    Train XGBClassifier and XGBRegressor exactly like scikit-learn estimators.

  2. Objectives and evaluation metrics · Essentials · 10 min

    Choose the loss XGBoost optimises (objective) and the metric it reports (eval_metric) for your task.

  3. Core hyperparameters · Essentials · 12 min

    Understand the handful of parameters that control model capacity, learning speed and randomness.

Data and the native API

  1. Missing values and categorical features · Essentials · 10 min

    Let XGBoost learn default directions for NaN and split natively on pandas categorical columns.

  2. Saving and loading models · Essentials · 7 min

    Persist trained models in XGBoost's portable JSON or UBJSON format and load them across versions.

  3. The native API: DMatrix, train and cv · Practical · 12 min

    Use xgb.DMatrix, xgb.train and xgb.cv for full control, custom callbacks and efficient data handling.

Training well

  1. Validation sets and early stopping · Practical · 10 min

    Monitor a validation set during training and stop adding trees when it stops improving.

  2. Tuning with scikit-learn tools · Practical · 12 min

    Search hyperparameters with RandomizedSearchCV or Optuna-style loops, using pipelines and sensible ranges.

  3. Feature importance and SHAP values · Practical · 12 min

    Explain XGBoost models with gain-based importance, built-in SHAP contributions and permutation importance.

Special tasks

  1. Imbalanced classification · Advanced · 10 min

    Handle rare positive classes with scale_pos_weight, sample weights, the right metric and threshold tuning.

  2. Monotonic and interaction constraints · Advanced · 11 min

    Force predictions to move in one direction with a feature, and restrict which features may interact.

  3. Learning to rank with XGBRanker · Advanced · 12 min

    Train ranking models for search and recommendations with query groups and NDCG-based objectives.

Internals and scale

  1. How XGBoost builds trees · Expert · 15 min

    Second-order gradient boosting, the regularised gain formula, histogram split finding and sparsity-aware splits.

  2. Custom objectives and metrics · Expert · 13 min

    Optimise your own loss by supplying gradients and hessians, and report custom evaluation metrics.

  3. GPUs, large data and distributed training · Expert · 12 min

    Train on GPUs with device='cuda', handle data larger than memory, and distribute with Dask or Spark.