Regularisation and tuning

Beyond num_leaves, a handful of parameters reliably trade training fit for generalisation.

LightGBM's anti-overfitting toolbox:

Row subsampling (bagging): subsample (bagging_fraction) below 1, together with subsample_freq (bagging_freq) of 1 or more. Bagging is off unless the frequency is positive, a common surprise.
Column subsampling: colsample_bytree (feature_fraction) 0.5-0.9; feature_fraction_bynode samples per split.
Penalties: reg_lambda (L2) and reg_alpha (L1) on leaf values, both 0 by default; min_split_gain requires a minimum gain per split.
Leaf constraints: min_child_samples, min_child_weight, max_depth, num_leaves.
Feature binning: smaller max_bin gives coarser splits, a mild regulariser.
path_smooth and extra_trees=True (random thresholds) add smoothing and randomness.

A practical recipe:

1. Fix learning_rate at 0.05-0.1 and use early stopping. 2. Tune num_leaves and min_child_samples. 3. Tune colsample_bytree, subsample with subsample_freq=1. 4. Tune reg_lambda, reg_alpha, min_split_gain. 5. Lower the learning rate (and raise the number of trees) for the final model.

Use random search or Bayesian optimisation (Optuna has a dedicated LightGBM tuner) rather than large grids, and remember that LightGBM uses all cores by default, so parallelise either the search or the model.

Example

import lightgbm as lgb
from sklearn.datasets import make_regression
from sklearn.model_selection import cross_val_score

X, y = make_regression(n_samples=2000, n_features=40, n_informative=10,
                       noise=30, random_state=0)
base = dict(n_estimators=400, learning_rate=0.05, verbose=-1, random_state=0)

for extra in [{},
              {"subsample": 0.7},                               # no effect: freq is 0
              {"subsample": 0.7, "subsample_freq": 1, "colsample_bytree": 0.7},
              {"subsample": 0.7, "subsample_freq": 1, "colsample_bytree": 0.7,
               "reg_lambda": 5.0, "min_child_samples": 40}]:
    m = lgb.LGBMRegressor(**base, **extra)
    print(round(cross_val_score(m, X, y, cv=5).mean(), 4), extra)
# 0.9056 {}
# 0.9056 {'subsample': 0.7}                 (identical: bagging never ran)
# 0.9224 {'subsample': 0.7, 'subsample_freq': 1, 'colsample_bytree': 0.7}
# 0.9264 {... 'reg_lambda': 5.0, 'min_child_samples': 40}

Remember

Bagging needs subsample_freq > 0 as well as subsample < 1
Feature fraction, L1/L2 and leaf constraints all reduce overfitting
Tune capacity first, then sampling, then penalties; lower the learning rate last

Watch out for

Setting subsample=0.8 without subsample_freq and believing bagging is on.

Check yourself

You set subsample=0.7 on LGBMRegressor and nothing changes. Why?

Official docs: lightgbm.readthedocs.io/en/stable/Parameters-Tuning.html#deal-with-over-fitting

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks